In an academic paper, Ermon's lab trained a GPT-2-sized transformer both autoregressively and as a diffusion model on identical data, and the diffusion version matched perplexity/quality while generating the same quality text with about 10x fewer neural network evaluations.
In a controlled AB-test-style comparison at GPT-2 scale, Ermon's lab showed a diffusion-trained model matched an autoregressive model's text quality but needed roughly 10x fewer network evaluations to generate it, the first demonstration that diffusion could work for text. ✦ AI generated
Stefano Ermon · The TWIML AI Podcast · 2026-03-26 · original ↗
starts at this moment · 11:29
“So what has been demonstrated to work for you text with diffusion?”
we showed that for the first time we were able to match the quality. So that in terms of perplexity in terms of like the quality of the text that these two models are able to generate it was about the same but the diffusion model was significantly faster like you could generate the same quality of text in about 10x less so 10 times less sort of like number of neural network evaluations.
verbatim transcript · starts at 11:29
11:29the first time it was possible to train a transformer based model so you basically took a GPT-2 size model and then you train it autoregressively the usual way you train it to predict the next token the way everybody else is training LLMs and you can train the same neural network as a diffusion model. And in that paper we showed that for the first time we were able to match the quality. So that
11:52in terms of perplexity in terms of like the the quality of the text that these two models are able to generate it was about the same but the diffusion model was significantly faster like you could generate the same quality of text in about 10x less so 10 times less sort of like number of neural network evaluations. So diffusion models were significantly more efficient at the GPT-2 scale. And
12:18so just so I understand the setup there are you saying you said train it autoregressively and train it via diffusion are you say is that two different models that you're comparing or are you sequentially training it autoregressively and then with diffusion? So it's a it's really just like a um almost like an AB test where you know it's a very very much a fair comparison in the sense that you
12:46take the same exact neural network architecture with the same number of parameters you train it on the same amount of data you just train it in on the one hand as a typical autoregressive model where you just predict the next token and that's how you use it at inference time and then on the other hand you can train it as a diffusion model and so at that point kind of like the
13:05difference in performance is entirely due to the different modeling paradigm diffusion versus autoregressive model. How did you overcome the discrete challenge in training that model? Yeah, so that was kind of like the the main kind of like idea in that paper like there were some new mathematics some new methods of basically uh figuring out what it means to do diffusion in the context of discrete text like objects