Simon downplays the idea that distillation is central to Chinese open-weight labs' progress. He argues environment-heavy RL processes cannot be distilled, and what actually powers progress is smart people, interesting algorithms, and data — sentiments Matt extends into a policy point that turning off distillation would not solve anything since the labs are genuine innovators. ✦ AI generated
Simon Mo · a16z Podcast · 2026-08-06 · original ↗
starts at this moment · 43:18
“Do you think distillation like is a critical component of what they they do or or like are they kind of just doing good work and and you know distillation if if it's done is sort of an incidental part of it.”
I I I would lean to the latter part specifically as I mentioned previously environment matters so much today. So these are RO environments right these cannot be distilled like you don't have other people's environment to really distill a copy from... you cannot distill how the model learns within environment a lot of these are just not doable today... I really don't think from currently what we're seeing uh this is a big cornerstone of what's powering the progress today. In the end what's powering the progress is still just um really smart people with very interesting algorithms, data environment...
verbatim transcript · starts at 43:18
42:59actually is better. Um, >> we we didn't um >> we didn't talk about distillation much so far in this conversation, but I think it's very relevant to this. Like >> I I have just one question which is like I'm I'm not going to ask like is distillation happening? I think this is kind of speculation on the part of everybody, you know, in the world, but like >> you work a lot with these Chinese labs.
43:18Do you think distillation like is a critical component of what they they do or or or like are they kind of just doing good work and and you know distillation if if it's done is sort of an incidental part of it. I >> I I would lean to the latter part specifically as I mentioned previously environment matters so much today. So these are RO environments right these
43:41cannot be distilled like you don't have other people's environment to really distill a copy from is about constructing it understanding also understanding the learning process you cannot distill how the model learns within environment a lot of these are just not doable today um there are things potentially you can do with rewriting the data sets right making better pre-training data but again you can do it with any models any models
44:08that are are going to follow instruction are going to be useful in terms of utility there. So I really don't think from currently what we're seeing uh this is a big cornerstone of what's powering the progress today. In the end what's powering the progress is still just um really smart people with very interesting algorithms, data environment and they will produce of course compute they will produce the models.
44:33>> I think it has really interesting policy implications. I I tend to agree with you by the way that that you know we have smart people everywhere working on a bunch of smart things and it's not about you know distilling data from any one place. um has really interesting policy implications, right? Because it doesn't, >> you know, it's almost tempting if you're if you're sort of, you know, in the
44:51White House to say, "Oh, sure. We'll just we'll just turn off distillation. All our problems will be solved." But like, >> you know, I think it's more the case that they're just, you know, smart people doing interesting things. And so it's so it's like how do we how do we kind of like adapt adapt to that? I think it's >> Yeah. And creative innovations, right? Like one part in my essay we kind of