ATRIUMsearch → argument graph
FactVideo · 31:20 — 32:50

Existing LLM serving engines like vLLM, SGLang, and TensorRT cannot run diffusion language models, so Inception had to build its own serving engine from scratch, since the diffusion LLM tooling ecosystem is far less mature than for autoregressive models.

Ermon explains that a major barrier to production diffusion LLMs is serving infrastructure: mainstream engines are built for autoregressive continuous batching and can't run diffusion models, forcing Inception to build its own serving stack while the open-source ecosystem (e.g., SGLang) only recently began adding support. ✦ AI generated

Stefano Ermon · The TWIML AI Podcast · 2026-03-26 · original ↗

Share card ↓Export .mdExport .txt

starts at this moment · 31:20

Elicited by

“How does the serving setup change for diffusion models?”

you cannot run a diffusion language model on existing serving engines. So if you think about vLLM, SGLang, TensorRT, uh these frameworks that exist... they're really really good at serving autoregressive LLMs very efficiently... The space for diffusion language models is much much less developed. So we had to build our own serving engine.

verbatim transcript · starts at 31:20

Transcript · around this moment

31:20challenging kind of like aspect. And I think one of the reasons why uh there are still no other providers that are able to serve diffusion language models uh in production today. Uh you cannot run a diffusion language model on existing uh serving engines. So if you think about vLLM, SGLang, TensorRT, uh these frameworks that exist and I know are even open source, and and they're really really good at serving uh

31:47autoregressive LLMs very efficiently. So they will handle things like continuous batching for you. Like when there is a stream of requests coming in, how do you batch them together to um to to to to serve them efficiently? And there is all kinds of optimizations that you need to do once you have access to multiple GPUs and many requests and um you know, there is a lot of existing

32:08kind of like frameworks and great work that has been done for autoregressive models. Uh the space for diffusion language models is much much uh less developed. So we had to build our own serving engine uh over the last maybe month or two, there's been some support for diffusion language models in SGLang uh for the open source models that have been open source diffusion language models that have been developed by the

32:33community. So there is starting to be a little bit of ecosystem, a little bit of tooling, a little bit of community support for diffusion language models in the in the open source community, but it's still uh you know, not nearly as developed as for autoregressive models. You talked earlier about the ability to change the number of refinement steps and kind of how powerful that is in um diffusion's own type of inference

33:01time scaling. Is that something that is currently uh you know, is it's you know, to I'm I'm thinking like on the static to dynamic spectrum, uh is it fully dynamic? Is it fully static? Is it per request static? Like how how do you think about uh that that um the the knobs there? Yeah, so it's a design choice. I think it you know, like to some extent

✳ SGLang✳ TensorRT✳ vLLM
Related moments