At smaller, specialized model scales, biased/structured components like gating mechanisms improve performance, but as models scale up they benefit from becoming maximally unstructured (pure attention, pure matrix multiplication)—architectural bias is a function of scale, not a universal good.

63:27is a extremely rich unbiased format of algorithms. Even if the computation complexity of attention is n to the power two, maybe we really need n to the power two to really get to that kind of level, you know, and maybe even we need like more complex architectures and we've always like tried to actually reduce the complexity of architectures for the sheer purpose of the fact that we are resource constraint as humanity

63:49as a whole. We are resource constraint right now, you know. So I would say the discoveries that we have right now it just shows that there is a gradient on architecture that you can follow as you scale models. The gradient that you're following is the fact that for smaller kind of models and specialized models, you can put as many biases like these gating mechanisms that you're bringing

64:12in, you know, and as me you can you can play around with a with as many operators of interest like in the in in in your computational graph, you know, and it is going to work and it is going to give you some sort of a boost if you're really maximizing for like linear um let's say linear time complexity like you want to implement linear attention systems, you know, like just the fastest

64:34kind of if the speed is like so important and you're actually wanting to even sacrifice a little bit quality, you can bring in like linearity and the whole system could be linear. You know, you don't even need some of those hybrids. hybrid. It just boosts that accuracy because as we see the as I said the O and2 like basically the complexity that that computition complexity at a

64:55certain level at a certain scale it is needed for us to really like get to those performances that you want to do. The larger the network becomes the more unstructured you can make it. That's kind of the learning from that that whole kind of algorithmic approach that we started designing neural architectures. >> Yeah, very interesting. Okay, let's look at the other end of the spectrum then as

65:16you work with customers. What are some interesting examples of when given resource constraints, given the narrowness of the domain of interest, other kinds of bias are actually winning in the architecture search process. >> Great question. So for example, if you go to let's say biology, you know, and you want to you want to model sequential data, you know, in biology, we're talking about DNA data in biology. DNA data from a vocabulary