A year and a half before Mamba, our Liquid-S4 paper first introduced input-dependent SSMs — the gating mechanism that later became central to Mamba and other sequence architectures originated in our liquid neural network research.

53:41structure you know that's when I say like this input dependent kind of thing that is actually coming into Mamba so right before Mamba actually got out you know like actually not right before but a year and a half before Mamba come come out we released a paper called liquid S4 you know this paper Because if you just read the abstract of that paper, you know, like we are actually for the first

54:02time we're introducing this idea of input dependent SSMs. Basically input dependent SSM. The idea there was that let's bring the the fundamental let's say building block that we we found like has a lot to do with representation learning with capacity of learning this input dependence kind of element to really bring that format of gating to the neur neural architectures to to to to SSM you know and then bringing that

54:30to convolutions as well with with the other kind of uh systems that we are designing today you know liquid foundation models as well that kind of format of gating It actually adds a lot of like you should you should just think about like naturally it makes a lot of sense if the neural networks it's not about the forward pass of the neural network that it is adaptive you know

54:49like because once when you're input dependent you have you know like input dependence in the learning from a learning theory perspective every all the magic happens in the backward pass you know if you think about when you're computing the gradients backwards that input dependent kind of operator itself it's going to represent itself So when you're learning from that data that you're seeing, you're also learning some sort of dynamics, input dependent

55:16dynamics there. So that that that second axis that I was talking about, it's not just the number of parameters of the neural networks is also the dynamics that the neural network actually learns. You have some representation of that dynamics being learned in the backward pass. Now the the the complexity of that gate is going to make the make a huge change. And also it turns out like that

55:36gate is extremely important also like in language modeling especially when you have sequence models like RNN's like uh let's say classical control like let's say continuous time RNN's let's say discretized RNN like LSDMs and then you have liquid neural networks like again not very nonlinear versions of these linear versions of recon is an SSM and and when you add this type of gating you you can improve the language