Weight-space learning enables knowledge transfer across domains that generates domain-specific models with 20-30x less compute than training from scratch, demonstrated by generating remote sensing foundation models from an ImageNet-based encoder-decoder.
Borth presents the remote sensing case study where weight-space learning generated remote sensing foundation models using only 350 GPU hours, compared to 12,000 GPU hours for the state-of-the-art, by transferring knowledge from ImageNet-trained models across domains. ✦ AI generated
Damian Borth · The TWIML AI Podcast · 2026-07-27 · original ↗
starts at this moment · 28:25
um therefore we had this paper that we're going to publish soon in CVPR about remote sensing where we take an imageet vit use our machinery to generate remote sensing models or remote sensing foundation models then we have the domain change where our encoder decoder is providing knowledge transfer to generate models that are better than the ones that imageet fine-tuning would be able to reach. So here we have a true knowledge transfer which is really really great where we are able actually to outperform or be you know equally in performance with current models like TFM published at ICLR I think the auto claim they trained for 12,000 GPU hours and we are able to do this on 350 GPU hours that's a factor of I don't know 20 25 30 depending on how you count and this is suddenly interesting because you train from models and not from data. So if you think about right we're running out of data that's the reason why the scaling laws are a little bit you know considered differently and everybody is moving into test time adaptation [clears throat] test time training we're running out of data to train the large models but we are not using the weights of older models so why not using the weights all the knowledge that all the compute that people invested right
verbatim transcript · starts at 28:25
28:25is leading to the situation that we need a well-trained model to generate another well-trained model which doesn't make sense if it's in the same domain because well I have a model why should I generate one >> chicken an egg problem right >> exactly um so therefore we had this paper that we're going to publish soon in CVPR about remote sensing where we take an imageet vit use our machinery to
28:49generate remote sensing models or remote sensing foundation models then we have the domain change where our encoder decoder is providing knowledge transfer to generate models that are better than the ones that imageet fine-tuning would be able to reach. So here we have a true knowledge transfer which is really really great where we are able actually to outperform or be you know equally in performance with current models like TFM
29:16published at ICLR I think the auto claim they trained for 12,000 GPU hours and we are able to do this on 350 GPU hours that's a factor of I don't know 20 25 30 depending on how you count and this is suddenly interesting because you train from models and not from data. So if you think about right we're running out of data that's the reason why the scaling
29:43laws are a little bit you know considered differently and everybody is moving into test time adaptation [clears throat] test time training we're running out of data to train the large models but we are not using the weights of older models so why not using the weights all the knowledge that all the compute that people invested right >> it's also an interesting context like if the if all the d if you know if we're in
30:05fact running out of data at least in you specific domains, visual text, etc. Then all that data is already in a bunch of models. Why replicate that and why not, you know, just find ways to different ways to slurp it out of the existing models? That's essentially the the premise, right? >> Totally makes sense. In the remote sensing community, we have 70 foundation models according to some surveys and and
30:30there are still people training the 71st, 72nd one, right? So why not take in this knowledge compress it all in in a weight space based learning representation and then sample models on demand because if you have a foundation model that is a vit with 800 900 parameters and somebody fine-tunes it to a task this person the partitioner needs to use all the 900 parameters and that's demanding compute and maybe the
- ·ImageNet ViT encoder-decoder transfers knowledge to remote sensing
- ·Outperforms ImageNet fine-tuning on domain-specific models
- ·True knowledge transfer across domains, not just adaptation
- ·Generates foundation models from prior weights, not from data
- ·TFM (ICLR) trained for 12,000 GPU hours from scratch
- ·Weight-space learning generated equivalent model in 350 GPU hours
- ·Matches or outperforms existing remote sensing foundation models
- ·Training data is running out — scaling laws are shifting
- ·Industry already invested massive compute into existing weights
- ·Reuse weights rather than discarding prior model knowledge