Ermon explains why porting diffusion from images to text was hard: pixel colors form a continuum you can interpolate between, but there's no natural 'in-between' for two words, so the mathematics built for continuous image spaces doesn't translate to discrete text. ✦ AI generated
Stefano Ermon · The TWIML AI Podcast · 2026-03-26 · original ↗
starts at this moment · 8:13
“You've mentioned discrete a couple of times as opposed to continuous. Can you talk about why that presents a challenge for diffusion models?”
if you think about text, and you take two words, then it's not clear what's in between the meaning of two different words, right? ... there is no real geometry to the space of possible tokens or possible words. And so, that makes the idea of denoising much more challenging because it's not clear what it means to perturb the noise to text.
verbatim transcript · starts at 8:13
8:13challenge for diffusion models? Yeah, of course. So, if you think about, you know, like a an image or just even a single pixel, you know, it's a you can think of it as a the bunch of colors and uh the the interesting thing is that if you change the colors a little bit, you know, the meaning doesn't change. So, in particular, you can kind of think about
8:34two possible colors for a pixel and all the kind of things in between them still make sense and they don't change the meaning of the image in any dramatic way, right? Uh but if you think about text, and you take two words, then it's not clear what's in between the meaning of two different words, right? Uh and so, there is no real geometry to the space of possible tokens or possible
8:59words. And so, that makes the idea of denoising much more challenging because there's it's not clear what it means to perturb uh the noise to to text. It's not clear how you build the like the whole geometry does not exist. And so, a lot of the concepts that were defined for um that that were invented to get diffusion models to work on images and video, they were kind of like relying
9:24very heavily on the fact that there is some kind of continuum of of of possible images and you can kind of like interpolate between them and it makes sense to get the model to kind of like smoothly move from one image to another. In the context of text and code, everything is very discrete and so, it's not obvious how you get the the mathematics that were developed for
9:48continuous spaces do not translate immediately to to discrete spaces. When you talk about the idea of words between points and, you know, words in a neighborhood it calls to mind and betting spaces and the like. You know to what degree I imagine that's been tried or you know and or maybe part of the ultimate solution to getting it to work for text. Yeah, so that can be
10:13there are approaches that essentially try to build the diffusion models for language generation kind of like in the embedding space so where you embed everything and you build a diffusion model and then the problem is that essentially you have to eventually decode back to text right eventually you cannot give an embedding to to to to your users or your customers and so that's always the problem that