As neural networks scale up, the optimal design becomes progressively less structured and less biased, which is why unstructured attention-based transformers dominate at massive scale while adding architectural biases badly hurts performance at that scale.

33:43functional kind of things that represent and it is unstructured. You can literally multiply any mat matrix into any matrix of any size. So the whole idea here is this. The larger neural networks that you make into infinite size. The larger neural networks you make the more want you want them to become less and less structured and we've seen like the success of transformers at the trillions of

34:08parameters. Now we are talking about tens of trillions of parameters like I mean the next generation of models that are going to come we're talking about trillions of and you can do that with a transformer architecture. As soon as you start adding a little bit of bias in that architecture at scale things become completely messed up. So we are talking about right now liquid neural networks like these alternative architectures

34:30that we we are talking about they have a scale to heat you know like there is a regime of parameters that you can just do better you know like let's say up to let's say 100 billion parameters up to a trillion parameter you know like that's kind of the range we are we are operating right now in this range the smaller is the model architecture the more you want to specialize them for a

34:52certain application to solve and they are actually like mathematically they're biased to actually solve a certain type of tasks better than any better than other types of architectures. So I would say biases on algorithms you know the more the more kind of biases you put on and what by biases what I mean is this adding a lot more nonlinearity adding like multiple gating levels on top of a

35:14neural network you know adding like recurrence you know like multiple different types of recurrence adding let's say convolutions like as a I mean convolution itself if you just keep the convolution or kind of neural networks they're also pretty unstructured because you can apply convolutions on any size of neural networks you know so that that that that question becomes like how much bias and what kind of problems you want

35:35to solve so that the bias in the neural neural architectures becomes kind of a function of I would say scale of the neural networks as well as like the the use cases you want to solve the most you see so the spectrum would be the more you go to the larger size the more unstructured kind of mathematical operators use pure matrix multiplication use pure convolutions use pure let's say