The optimal amount of architectural bias (gating, nonlinearity, structure) in a neural network is inversely related to model scale: smaller, specialized models benefit from more biased/structured operators, while at very large scale (up to trillions of parameters) unstructured, unbiased architectures like pure attention and matrix multiplication win.

34:08parameters. Now we are talking about tens of trillions of parameters like I mean the next generation of models that are going to come we're talking about trillions of and you can do that with a transformer architecture. As soon as you start adding a little bit of bias in that architecture at scale things become completely messed up. So we are talking about right now liquid neural networks like these alternative architectures

34:30that we we are talking about they have a scale to heat you know like there is a regime of parameters that you can just do better you know like let's say up to let's say 100 billion parameters up to a trillion parameter you know like that's kind of the range we are we are operating right now in this range the smaller is the model architecture the more you want to specialize them for a

34:52certain application to solve and they are actually like mathematically they're biased to actually solve a certain type of tasks better than any better than other types of architectures. So I would say biases on algorithms you know the more the more kind of biases you put on and what by biases what I mean is this adding a lot more nonlinearity adding like multiple gating levels on top of a

35:14neural network you know adding like recurrence you know like multiple different types of recurrence adding let's say convolutions like as a I mean convolution itself if you just keep the convolution or kind of neural networks they're also pretty unstructured because you can apply convolutions on any size of neural networks you know so that that that that question becomes like how much bias and what kind of problems you want

35:35to solve so that the bias in the neural neural architectures becomes kind of a function of I would say scale of the neural networks as well as like the the use cases you want to solve the most you see so the spectrum would be the more you go to the larger size the more unstructured kind of mathematical operators use pure matrix multiplication use pure convolutions use pure let's say

35:59sequential scan pure pure pure operators. You see, it's like you you get you get m much more like and then forget about gates, forget about like adding like you know like fancy like uh uh control like forget gates and all those things and at scale this is something that we we've we have seen like because we we have scaled like neural networks and we have like a