The curse of dimensionality is only tractable because the world is hierarchical: if data has hidden coarse-grained variables, deep architectures are particularly good at discovering them with polynomially many data rather than exponentially many.
Wyart explains that the exponential volume of high-dimensional data makes interpolation hopeless unless the data has hierarchical structure. The manifold hypothesis is insufficient because kernel methods and shallow networks would then work perfectly, yet they fail. The real answer is that deep architectures discover hidden hierarchical abstractions, reducing dimensionality polynomially. ✦ AI generated
Matthieu Wyart · Machine Learning Street Talk · 2026-08-10 · original ↗
starts at this moment · 48:43
“But you're saying that that this kind of emerging behavior is actually how it is tractable.”
So what it means is that even if you give me one trillion points, because the volume is so huge, they're extremely far away from each other. Extremely far away. And so if you have a machine that's just interpolating and now you ask a question about a new test point and you can prove mathematically that if the data has little structure like you're trying to learn to regress some function that's smooth it's hopeless. I mean the only way you will extrapolate and have you know power to to to generalize is if you know you bring those point together. it means you have an exponentially large number of data. You have more data than atoms in the universe. So it's just impossible. So to me this is completely fundamental question and um and it's true that sometimes in the literature it's tossed aside by saying okay it's super naive to say that the dimension is a number of pixel on an image. In fact it should really be uh that the data lie in a lower dimension manifold. And if you try to measure it, it's true that it lies in a lower dimension manifold, but this dimension is still large in dimen in dimension. And and to me the big problem is that if this was the answer to this question, it will mean that very simple algorithm like kernel methods are ancestor of deep net or even shallow network. I mean they do it perfectly. And if you give them you know a low dimensional manifold you don't need to have any interesting architecture but if you use zoo on text I can tell you it fails lament I mean it's completely does nothing
verbatim transcript · starts at 48:43
48:34about this. He's got this spline theory of neural networks and he said that you know in high dimensions all data is extrapolation. There's no manifold. That's It's actually doing this spline decomposition in an input sensitive way. Lots of lots of people have different ideas about this. Yes. >> But you're saying that that this kind of emerging behavior is actually how it is tractable. >> Exactly. So uh actually that's before we
48:58started thinking about creativity our first work was really trying to understand what what you know sort of data structure allow deep net to actually perform. So it's exactly as you said maybe I can say it again. So in physics we know that a volume goes like a length to the exponent of the dimension. So in 3D L cube in 2D L² L is a length. So think about a large
49:22dimension. So if you think about an image, D may be the number of pixels at least naively. If you think about text, it may be the number of words in your sentence again naively. So those volumes are huge. They're exponential. You know, they're exponential at large in the dimension. So what it means is that even if you give me one trillion points, because the volume is so huge, they're
49:47extremely far away from each other. Extremely far away. And so if you have a machine that's just interpolating and now you ask a question about a new test point and you can prove mathematically that if the data has little structure like you're trying to learn to regress some function that's smooth it's hopeless. I mean the only way you will extrapolate and have you know power to to to generalize is if you know
50:16you bring those point together. it means you have an exponentially large number of data. You have more data than atoms in the universe. So it's just impossible. So to me this is completely fundamental question and um and it's true that sometimes in the literature it's tossed aside by saying okay it's super naive to say that the dimension is a number of pixel on an image. In fact
50:36it should really be uh that the data lie in a lower dimension manifold. And if you try to measure it, it's true that it lies in a lower dimension manifold, but this dimension is still large in dimen in dimension. And and to me the big problem is that if this was the answer to this question, it will mean that very simple algorithm like kernel methods are ancestor of deep
51:02net or even shallow network. I mean they do it perfectly. And if you give them you know a low dimensional manifold you don't need to have any interesting architecture but if you use zoo on text I can tell you it fails lament I mean it's completely does nothing so um so the question is why do you need deep architectures you know I mean and uh and some of the
51:28thing you said is it does not answer that that question so that's really the question we're we're looking after and so essentially The answer to that in in is that if the world is thearchical, if it has those hidden co grand variables, those machines are super good to discover them and they can discover them with a generically with an number of data that's not huge polomial in the
- ·One trillion points stay extremely far apart in high-dim
- ·Smooth-interpolating machines cannot generalize
- ·Coarse hidden variables are the only escape
- ·Deep nets find hierarchy with poly, not exp, data