Weight-space learning can be scaled beyond homogeneous model zoos to train on diverse open-weight models from Hugging Face across different architectures, tasks, and modalities.

20:54train and that's the vision right can you train a different sized models with different architectures tasks modalities from open weight repositors at hugging face could we download everything from hugging phase and train a foundation model of neuronet networks >> and you've you've started down that path I think the >> uh that rabbit hole >> the poster that that rabbit hole the poster that I originally saw saw at GTC

21:22that led me to you was something about kind of training on the hugging space model zoo. Right. >> Exactly. So um after we were able to scale up where we're thinking like can we kind of because we're still limited to the model zoo. Right. So so like like I I can tell you I can generate now a new neural network but I need this 1,000 neural networks to have trained before I

21:43can generate that one. So you tell me that's great but you know now we have 101 neuronet networks. So what what are we gaining right at the end of what are we winning? So that the next step was can we get can we train on models that are out there and there's amazing work on analyzing how you know hugging face looks like the model atlas from Yahoo

22:04and Yetit and there is a lot of there are a lot of models so can we download those models and then independently of what kind of architecture they have what data set they trained use our machinery and it's it's a little bit tricky because you know different sequence lengths different types of neural network layer. So we have to put some information into it. But we were able to

22:25train like the first um you know weight space learning model that can do generation and analysis of weights. So discriminative and generative downstream dust on hugging face models. So this was work by by D F which is really amazing. Um again we we thought it's much more challenging to do this but you know you have to scale you need the machinery and you need those little tricks how to

22:53handle those different you know the tokenizer needs to be adapted to arbitrary architectures that's that's the thing. Yeah. And what's the filter that needs to be applied on the you know this hugging face is vast library of models that normalizes them to something you can deal with. But first of all and this is work also that you know other figured out likely Yahoo there's a a lot

23:15of content on hugging face is not documented around 30% of the models they don't have any meaningful meta data so you don't know what what yeah I mean hugging face >> so the first filter just get rid of all of those >> yeah [laughter] knowing which models are helpful so we need we did a little bit of experiments if we scale scaling alone doesn't help you need to increase the diversity of

23:38the models So we want to have diverse models. So we want to have you know different data sets mostly focus on computer vision. Uh language is the next and uh we kind of developed a scoring function on you know how popular is the model how how uh is it apparent or is it some derived work? There's a lot of trees in there to download a a set of two we have 20,000