Pure scaling of models and data will not produce scientific creativity that discovers new subspaces, because such creativity requires interaction with the world, observing the intriguing, simplifying geometry, and building models — abilities machines trained on all of written text alone do not have.
Wyart distinguishes composing existing ideas from transformative creativity. While machines can compose new faces by learning rules of combination, scientific discovery — detecting what's intriguing, simplifying geometry, modeling — requires embodied interaction with the world. Scaling up on all written text alone will not yield this; we need to teach machines to be good scientists. ✦ AI generated
Matthieu Wyart · Machine Learning Street Talk · 2026-08-10 · original ↗
starts at this moment · 36:55
“yeah I think I'm just trying to understand what the gap is because it see it would be consistent with your argument If if you're saying that we can learn the abstract structure of the world and be um generatively competent in one domain, why why would we not? Because for me, creativity is not just about coherence and respecting the constraints. I mean transformative creativity in my mind is about discovering interesting new subspaces.”
I think creativity can be much more than that. I mean if we think about what we discussed about what it means to be a physicist and how science proceeds. I mean it's an example of creativity you know if you think about creativity like Newton understandings of motion of planets and things like that. I mean we talked about dialogue between experiments and theory. We talked about building models at a good level of description. We talked about you know analogies and I don't think I agree with you that I for example all that I don't think it's in the machine. Uh I think there's no reason why we would not be able one day to build machines that can do that. I'm not sure if just scanning up things will lead to that. Uh I think maybe we need to do more introspection of how we function as scientists to come up with a good data set and the good procedures to teach machines to be goodi good scientists.
verbatim transcript · starts at 36:55
36:35aren't good enough yet when they do understand you know when they have a theory of mind and they understand how the world works even more abstractly eventually we could just autonomously create coding agents that will just make Microsoft Word or something and and there is a path to that level of competence >> composition. So essentially what we're saying is that imagine that you have this diffusion model that learns to
36:55compose new faces. Essentially what it's doing is that when it has seen enough of low-level features like nose, eyes and mouth. It understands the rules of the games and it composes them together. Uh and what we also liked about this description is that we can make nonal prediction that you know we test with real images or with real text and maybe we'll come back to that because I think
37:17it's a really important part of physics. It's not creativity is not just putting pieces together that satisfy constraint. Although when you have a new idea often it's putting existing ideas together into a new hole. Uh but I think creativity can be much more than that. I mean if we think about what we discussed about what it means to be a physicist and how science proceeds. I mean it's an example of creativity you
37:46know if you think about creativity like Newton understandings of motion of planets and things like that. I mean we talked about dialogue between experiments and theory. We talked about building models at a good level of description. We talked about you know analogies and I don't think I agree with you that I for example all that I don't think it's in the machine. Uh I think there's no reason why we would not be
38:08able one day to build machines that can do that. I'm not sure if just scanning up things will lead to that. Uh I think maybe we need to do more introspection of how we function as scientists to come up with a good data set and the good procedures to teach machines to be goodi good scientists. They're just an example like creativity in science requires a lot of abilities uh to create something
38:30really new and how to interact with the world around us that you know I don't think machines have and yet I mean u so I think I agree with you that just scaling up I don't think will lead total success we need to you know have develop other abilities in those machines >> yeah I think I'm just trying to understand what the gap is because it see it would be consistent with your