We built an automated, hardware-in-the-loop evolutionary architecture search system (AFMD) specifically to remove human bias from foundation model design, since even top labs rely on small groups of researchers making subjective architecture calls.

42:44optimizing for memory consumption on that device, optimizing for latency, optimizing for speed. While no sacrifice on quality when we talk about quality, perplexity is not the measure. It's actually the application the downstream applications that we care about. You know, it's not just also public benchmarks. We're talking about 100 different benchmarks. So the problem space becomes from a metal learning perspective becomes like a very very complex kind of problem. Now I'll tell

43:09you that why did we take this approach you know like to design an architecture? Why? because we wanted to remove all the human biases early on as we are actually like building architectures. One of the things that we realize like culturally at companies like this is what I can tell you like even at largest foundation model labs uh in in the US right now entropic and open AI there are a bunch

43:29of people like people that are coming from the science like I call them the avengers of the architectures or avengers of the you know like post training or or or let's say pre-training these groups of people there's usually like a very small set of people that are calling the shots on like oh you know what you're going to tweak this portion of this architecture so that it performs

43:50better. Why? Because in my personal experiences, it has started working better. If you're really truly think and this is something that is broken in all the foundation model labs. You cannot say that like somebody has a fix to this. But now the recursive self-improvement kind of process is actually fixing for that because now people are just finally realizing you got to give it to the algorithms. You

44:10know, you have to be bitter lessons. You have to you have bitter lesson people. So you got to you you you got to be giving it to a systematic way to actually find out what is the true architecture for the for the problems that you want to solve. You can build like a general purpose computer. The insights that I shared with you in in in in the format of the scaling laws of

44:30neural networks is coming out of our massive exploration of the space of architectures. You know the fact that in the smaller kind of category of models the some biases on the architecture help in the larger instances of the models you don't need to bias the systems you can actually go pure convolutions pure transformers and you can go pure let's say recurrences that are like very simplified you don't need to add any