Algorithms that predict in abstract latent space rather than raw token space are significantly more sample-efficient because they build abstractions from stronger, less noisy signals.
Joint embedding prediction architectures — where a student network predicts the teacher's latent representations of occluded data rather than predicting raw tokens — are much more sample-efficient because they build abstract concepts by predicting nearby abstract concepts rather than correlating abstractions with noisy low-level pixels. ✦ AI generated
Matthieu Wyart · Machine Learning Street Talk · 2026-08-10 · original ↗
starts at this moment · 53:22
“this is very pertinent because you've got a paper out basically saying that we should predict in the in the latent space not the token space and his his idea essentially is that if we actually predict in latent space then we can be significantly more sample efficient than if than if we predict in in the ambient space.”
in those models what we found is that those algorithms that are introspective that learn from their own latent are much more powerful in terms of sample complexity and they will eventually they will learn the same the same abstraction but much faster to build abstraction You need to bring configuration. Think again about the street. All those configuration you need to understand it's one entity a street and your what diffusion or next token prediction do is that they the signal they have to bring those together has to do with pixels around very lowle features and I told you that this correlation between abstract things and things that are not abstract it's it's it's there it exists but it's very noisy but imagine instead that you when you start when you understood the concept of cars and passerby And you also understood the concept of house houses. Then essentially what those methods do is they can build the concept of street by predicting that those configuration have not pixel the painting of the house but just the concept houses nearby and then the signal is much larger and so you need much less data to extract the signal from noise.
verbatim transcript · starts at 53:22
53:22the brain, you know, learns languages with much less data than machines. So machines are amazing. They speak better English than me for for sure. But in some definition of intelligence, they need many more data than us to do those tasks. So u so what's why are we so different? Um and so something I mean there are many hypothesis but one thing that's discussed in the field is the
53:48fact that you know those large language models at the end they do something that's seems a bit trivial. It's like uh you mask a token and you try to uh to discover it. Even to do that in our models you find that you need to dis if you want to do it well you need you need to understand the full the full hierarchical abstraction of the world.
54:10even to do that. Well, actually we so that we started to work on next token prediction because I always had for at least one decade this sort of question related to the curse of dimensionality which was you know how come when we when we produce speech think about the end of a sentence maybe I said 30 words before the number of possible sentences is huge. How do I do I need to memorize
54:34those 30 words to do that? I mean how is it possible? And actually those models gave a sort of elegant answer to that because what happens when you try to break the next token is that you will if you said a long sentence maybe you would have you would have a coarse grand variable that describe a coarse meaning of the first half of the sentences and as you approach what you're going to say
54:55you have a finer and finer more precise description. So so so at least to me this sort of way of thinking led to a possible solution for my paradox. Um but so in any event uh even if you try to learn the next token you need to to to build uh those very those abstraction but I told you that one problem with doing this is that if you're very
55:19abstract it needs a lot of data because you build those abstraction by bringing together configuration in the data that predict a similar surrounding but at a low level like similar pixels around. So going back so so um what has been proposed in the literature actually it's interesting also in neuroscience there is this notion that maybe the brain is doing some sort of very interesting self-supervised learning where instead
55:47of just predicting what's going to be the next frame on its eyes. It's trying to predict the next activity of its cortex. So predicting in some sort of abstract space and these ideas also emerged in in machine learning and you talked about Yanluka and and there are also other models and they're extremely interesting uh and again the idea is instead of predicting at the level of token can I
56:13predict in more abstract space and they developed this field develop very interesting machines to do that I mean you can think about twins so you you have one machine you duplicate it and one machine is shown the entire data. It's a teacher and one machine is shown some occluded version of the data and your student has to predict not the tokens that were occluded but how those
- ·Predicting latent representations beats raw token prediction
- ·Models learn identical abstractions with far less data
- ·Sample complexity significantly improved
- ·Abstract-to-abstract learning is the key advantage
- ·Raw tokens: abstract concepts ↔ noisy pixel-level features
- ·Latent space: abstract concepts ↔ abstract concepts
- ·Signal magnitude much larger in latent prediction
- ·Cleaner signal → faster abstraction formation
- ·Traditional: correlate 'street' with raw pixel patterns
- ·Latent method: predict nearby 'house' concept, not pixels
- ·Abstractions built from other abstractions
- ·Much less data needed to learn higher-level concepts