Hassani explains that the core technical bottleneck in scaling liquid neural networks is that nonlinear relationships resist being reshaped into parallel tensor/matrix computations, which is why alternative architectures like state space models chose linear dynamics. ✦ AI generated
Ramin Hassani · The Cognitive Revolution · 2026-07-04 · original ↗
starts at this moment · 28:02
“So the obvious bitterpilled question would be... what if we take that exact paradigm and go to millions billions but what you must be hitting some bottlenecks along the way what are they?”
The main challenge is turning sequential computation into parallel computation. So when you have degree like when you start like going from a single neuron dynamics to multiple neural dynamics weight parameters of your system instead of being a scalers or vectors they become matrices and tensors. So now you're talking about matrix multiplication.
verbatim transcript · starts at 28:02
28:02some bottlenecks along the way what are they >> so the main challenge is turning sequential computition into parallel computation. So when you have degree like when you start like going from a single neuron dynamics to multiple neural dynamics weight parame like parameters of your system in instead of being a scalers or vectors they become matrices and tensors. So now you're talking about matrix multiplication. You want to turn the
28:34scalar computations into tensor computations, right? You want to be able to this is how you parallelize kind of let's say sequential computers. The problem is that not all the time like if you have nonlinear relationships in an equation, you cannot trivially create a onetoone map between a vectorzed computation and a tensorized computation. So those nonlinear kind of nonlinear attributes of liquid neural networks and any other nonlinear
29:03recurrent neural network I mean recurrence itself is adds like some degrees of complexity into into the math of the whole equation. I mean let alone if the if the recurrence has has a nonlinearity on it becomes like a lot more complex to disentangle vectors or write them with with with typical linear algebra kind of methods that we know into tensors. And you have to be able to
29:26turn these computations into tensors to be able to compute in parallel, you know, like you compute them once at a time. You see? So that's that's kind of the the the mathematical. It's not rel related only to liquid run. Any mathematical operation that you want to tensorize, if it is nonlinear, you're going to have the troubles that we're talking about. That's why the state space models that actually came out,
29:50these are all about around linear dynamics. You know, we're talking about like linear dynamical systems, right? And why those linear dynamical systems are important is just pure fact that we do not have proper ways to scale nonlinear systems. You can apply nonlinearity to a to a linear dynamical systems. For example, in a let's say like a gated or let's say you like you can you can you can add a sigmoidal