Video · 2026-03-26 · 1h 3m · 6 moments

The Race to Production-Grade Diffusion LLMs [Stefano Ermon] - 764

✦ AI generated

timeline · colored by role

Claim◆

Diffusion language models scale better than autoregressive models at inference time — they're cheaper to serve, faster, and yield more tokens per GPU, which lowers the price per token.

Stefano Ermon opens by arguing that diffusion LLMs beat autoregressive models on inference-time economics — lower price per token and lower watts per token — which is why Inception bet on the approach now that production economics matter most.

transcript

Stefano Ermon: what we're seeing with diffusion language models is that they scale better than auto-regressive models at inference time. They're cheaper to serve, they're faster, you get more tokens per GPU, which means that the price is actually lower.

gives example · 1supports · 1

Mechanism

Diffusion models generate images by starting from pure noise and iteratively denoising them, and this denoising training objective is much more stable to optimize than the adversarial game used to train GANs.

Ermon explains the origin of diffusion models: instead of the unstable two-network adversarial setup of GANs, a single network is trained on the simple, stable task of removing noise from images, which can then be run repeatedly at inference time to generate images from pure noise.

transcript

Stefano Ermon: we came up with this alternative approach which is now called a diffusion model where essentially you generate an image by starting from noise and then iteratively refining it until you get a crisp kind of like nice image that is consistent with the prompt at the end. And the key benefit is that the training objective is very stable. The neural network is trying to just denoise an image.

Mechanism

Text is discrete and has no meaningful geometry between tokens the way pixel colors have a continuum between them, which makes it unclear what it even means to 'add noise to' or denoise text, unlike images.

Ermon explains why porting diffusion from images to text was hard: pixel colors form a continuum you can interpolate between, but there's no natural 'in-between' for two words, so the mathematics built for continuous image spaces doesn't translate to discrete text.

transcript

Stefano Ermon: if you think about text, and you take two words, then it's not clear what's in between the meaning of two different words, right? ... there is no real geometry to the space of possible tokens or possible words. And so, that makes the idea of denoising much more challenging because it's not clear what it means to perturb the noise to text.

Data

In an academic paper, Ermon's lab trained a GPT-2-sized transformer both autoregressively and as a diffusion model on identical data, and the diffusion version matched perplexity/quality while generating the same quality text with about 10x fewer neural network evaluations.

In a controlled AB-test-style comparison at GPT-2 scale, Ermon's lab showed a diffusion-trained model matched an autoregressive model's text quality but needed roughly 10x fewer network evaluations to generate it, the first demonstration that diffusion could work for text.

transcript

Stefano Ermon: we showed that for the first time we were able to match the quality. So that in terms of perplexity in terms of like the quality of the text that these two models are able to generate it was about the same but the diffusion model was significantly faster like you could generate the same quality of text in about 10x less so 10 times less sort of like number of neural network evaluations.

Claim◆

Inception's newly launched Mercury 2 diffusion language model matches the quality of the best speed-optimized frontier-lab models (Haiku, Flash, Mini-class models) while being roughly 5-10x faster.

Ermon states that Inception's commercial-scale diffusion LLM, Mercury 2, has closed the quality gap with frontier labs' fast/small models like Haiku and Flash, while delivering answers 5-10x faster.

transcript

Stefano Ermon: the latest model that we announced this week Mercury 2 is actually matching in quality some of the best speed optimized models from frontier labs so we'll think about the Haiku models the flash models mini models from OpenAI... it's about 5 to 10x faster in terms of like the time it takes you to get an answer using a diffusion model versus an autoregressive model.

Fact

Existing LLM serving engines like vLLM, SGLang, and TensorRT cannot run diffusion language models, so Inception had to build its own serving engine from scratch, since the diffusion LLM tooling ecosystem is far less mature than for autoregressive models.

Ermon explains that a major barrier to production diffusion LLMs is serving infrastructure: mainstream engines are built for autoregressive continuous batching and can't run diffusion models, forcing Inception to build its own serving stack while the open-source ecosystem (e.g., SGLang) only recently began adding support.

transcript

Stefano Ermon: you cannot run a diffusion language model on existing serving engines. So if you think about vLLM, SGLang, TensorRT, uh these frameworks that exist... they're really really good at serving autoregressive LLMs very efficiently... The space for diffusion language models is much much less developed. So we had to build our own serving engine.

Highlight slides

Diffusion LMs beat autoregressive at inference economics✦ from: Diffusion language models scale better than autoregressive models at inference time — they're cheaper to serve, faster, and yield more tokens per GPU, which lowers the price per token.Why Inception bet on diffusion✦ from: Diffusion language models scale better than autoregressive models at inference time — they're cheaper to serve, faster, and yield more tokens per GPU, which lowers the price per token.Mercury 2 matches quality of frontier speed-optimized models✦ from: Inception's newly launched Mercury 2 diffusion language model matches the quality of the best speed-optimized frontier-lab models (Haiku, Flash, Mini-class models) while being roughly 5-10x faster.Mercury 2 delivers answers 5–10x faster✦ from: Inception's newly launched Mercury 2 diffusion language model matches the quality of the best speed-optimized frontier-lab models (Haiku, Flash, Mini-class models) while being roughly 5-10x faster.

Related episodes

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→

Video · a16z Podcast

Is Defense the Next Trillion-Dollar Category? | a16z American Dynamism Summit

→

Video · BG2 Pod

The SpaceX IPO, Fable 5, AI Capex Update & Market Check w/ Gavin Baker, Andrew Fox & Clark Tang

→