Deep networks solve the curse of dimensionality by discovering hierarchical abstract variables in data, allowing them to generalize with polynomial rather than exponential amounts of data.
When data has a hierarchical generative structure, deep networks can discover the underlying coarse-grained variables efficiently, reducing effective dimensionality and enabling generalization far beyond what shallow models or kernel methods can achieve. ✦ AI generated
Matthieu Wyart · Machine Learning Street Talk · 2026-08-10 · original ↗
starts at this moment · 48:58
“this curse of dimensionality. So you know there's always been this statistical law essentially that when we have high dimensions the number of data that you need you know to to to make it tractable increases exponentially.”
So in physics we know that a volume goes like a length to the exponent of the dimension. So in 3D L cube in 2D L² L is a length. So think about a large dimension. So if you think about an image, D may be the number of pixels at least naively. If you think about text, it may be the number of words in your sentence again naively. So those volumes are huge. They're exponential. You know, they're exponential at large in the dimension. So what it means is that even if you give me one trillion points, because the volume is so huge, they're extremely far away from each other. Extremely far away. And so if you have a machine that's just interpolating and now you ask a question about a new test point and you can prove mathematically that if the data has little structure like you're trying to learn to regress some function that's smooth it's hopeless. I mean the only way you will extrapolate and have you know power to to to generalize is if you know you bring those point together. it means you have an exponentially large number of data. You have more data than atoms in the universe. So it's just impossible. So to me this is completely fundamental question... and so essentially The answer to that in in is that if the world is thearchical, if it has those hidden co grand variables, those machines are super good to discover them and they can discover them with a generically with an number of data that's not huge polomial in the dimension once again.
verbatim transcript · starts at 48:58
48:58started thinking about creativity our first work was really trying to understand what what you know sort of data structure allow deep net to actually perform. So it's exactly as you said maybe I can say it again. So in physics we know that a volume goes like a length to the exponent of the dimension. So in 3D L cube in 2D L² L is a length. So think about a large
49:22dimension. So if you think about an image, D may be the number of pixels at least naively. If you think about text, it may be the number of words in your sentence again naively. So those volumes are huge. They're exponential. You know, they're exponential at large in the dimension. So what it means is that even if you give me one trillion points, because the volume is so huge, they're
49:47extremely far away from each other. Extremely far away. And so if you have a machine that's just interpolating and now you ask a question about a new test point and you can prove mathematically that if the data has little structure like you're trying to learn to regress some function that's smooth it's hopeless. I mean the only way you will extrapolate and have you know power to to to generalize is if you know
50:16you bring those point together. it means you have an exponentially large number of data. You have more data than atoms in the universe. So it's just impossible. So to me this is completely fundamental question and um and it's true that sometimes in the literature it's tossed aside by saying okay it's super naive to say that the dimension is a number of pixel on an image. In fact
50:36it should really be uh that the data lie in a lower dimension manifold. And if you try to measure it, it's true that it lies in a lower dimension manifold, but this dimension is still large in dimen in dimension. And and to me the big problem is that if this was the answer to this question, it will mean that very simple algorithm like kernel methods are ancestor of deep
51:02net or even shallow network. I mean they do it perfectly. And if you give them you know a low dimensional manifold you don't need to have any interesting architecture but if you use zoo on text I can tell you it fails lament I mean it's completely does nothing so um so the question is why do you need deep architectures you know I mean and uh and some of the
51:28thing you said is it does not answer that that question so that's really the question we're we're looking after and so essentially The answer to that in in is that if the world is thearchical, if it has those hidden co grand variables, those machines are super good to discover them and they can discover them with a generically with an number of data that's not huge polomial in the
51:53dimension once again. And once they discover them, it's a sort of summary of what the data is, you know, instead of describing pixel by pixel. Oh, is there a nose, ears, and so on. So you're reducing the dimension of the problem essentially and you can solve the curse of dimensionality. So I think this explanation has the advantage that it I mean whatever you explanation you come up with it has to explain it needs to
- ·Volume grows exponentially with dimension
- ·Data points become extremely far apart
- ·Even one trillion points insufficient for interpolation
- ·Shallow models cannot generalize without exponential data
- ·Hierarchical data has hidden coarse-grained variables
- ·Deep networks discover these variables efficiently
- ·Reduces effective dimensionality dramatically
- ·Generalization requires only polynomial data