Predicting in latent or abstract space rather than token space is much more sample-efficient for learning abstractions, because building a coarse-grained concept by correlating with other abstract concepts gives a much stronger signal than correlating with low-level pixels or tokens.
The core thesis of the paper: introspection-based algorithms that learn from their own latent representations are much more powerful in sample complexity. When building a coarse-grained concept like 'street', correlating with already-understood abstract concepts (houses) gives a far stronger signal than correlating with noisy low-level pixels, so far less data is needed. ✦ AI generated
Matthieu Wyart · Machine Learning Street Talk · 2026-08-10 · original ↗
starts at this moment · 52:24
“he's got this idea called a joint embedding prediction architecture and this is very pertinent because you've got a paper out basically saying that we should predict in the in the latent space not the token space and his his idea essentially is that if we actually predict in latent space then we can be significantly more sample efficient than if than if we predict in in the ambient space.”
imagine instead that you when you start when you understood the concept of cars and passerby Uh and you also understood the concept of house houses. Uh then essentially what those methods do is they can build the concept of street by predicting that those configuration have not pixel the painting of the house but just the concept houses nearby and then the signal is much larger and so you need much less data to extract the signal from noise. So yes we do find that those to understand the archical structure of the world those at least in those simple models it's much more efficient.
verbatim transcript · starts at 52:24
52:16explain why you need deep networks. >> Yes. Yes. You know when when I was speaking with Tom from Goodfire the other day he was saying that so much of interpretability is going from um essentially neural representations to text you know to try and introspect about them. and and he thinks we could have a new type of training method where we go from text to back to neural
52:36representation. So we we we discover these um you know emerging modular structures and during training we we kind of encourage them to be even more pristine even more evolved but there are other folks talking about similar ideas as well. So Yan Yan Lan for for example he's got this idea called a joint embedding prediction architecture and this is very pertinent because you've got a paper out basically saying that we
52:57should predict in the in the latent space not the token space and his his idea essentially is that if we actually predict in latent space then we can be significantly more sample efficient than if than if we predict in in the ambient space. Tell me about that. [snorts] >> Yes. So I Okay. So that's a question we've been uh fascinated by in the last year or two. Um as as we just discussed,
53:22the brain, you know, learns languages with much less data than machines. So machines are amazing. They speak better English than me for for sure. But in some definition of intelligence, they need many more data than us to do those tasks. So u so what's why are we so different? Um and so something I mean there are many hypothesis but one thing that's discussed in the field is the
53:48fact that you know those large language models at the end they do something that's seems a bit trivial. It's like uh you mask a token and you try to uh to discover it. Even to do that in our models you find that you need to dis if you want to do it well you need you need to understand the full the full hierarchical abstraction of the world.
54:10even to do that. Well, actually we so that we started to work on next token prediction because I always had for at least one decade this sort of question related to the curse of dimensionality which was you know how come when we when we produce speech think about the end of a sentence maybe I said 30 words before the number of possible sentences is huge. How do I do I need to memorize
54:34those 30 words to do that? I mean how is it possible? And actually those models gave a sort of elegant answer to that because what happens when you try to break the next token is that you will if you said a long sentence maybe you would have you would have a coarse grand variable that describe a coarse meaning of the first half of the sentences and as you approach what you're going to say
54:55you have a finer and finer more precise description. So so so at least to me this sort of way of thinking led to a possible solution for my paradox. Um but so in any event uh even if you try to learn the next token you need to to to build uh those very those abstraction but I told you that one problem with doing this is that if you're very
- ·Predicting in latent/abstract space beats token space for learning abstractions
- ·Correlating with abstract concepts (houses) gives far stronger signal than pixels
- ·Coarse concepts like 'street' built from already-understood abstractions
- ·Much less data needed to extract signal from noise