Speculative decoding speeds up LLM inference by having a small draft model predict several tokens ahead, then having the large model verify them all in one batched pass — which is faster because batched verification is compute-bound rather than limited by memory bandwidth like standard one-token-at-a-time generation.
Akshat explains the mechanics behind speculative decoding: a smaller draft model proposes tokens ahead of time and the larger model verifies them in a batch, escaping the memory-bandwidth bottleneck of normal decoding. ✦ AI generated
Akshat Bubna · Latent Space · 2026-07-08 · original ↗
plays this moment only · 16:52 — 18:22
“Anything you wanna point out from this around, what people should know?”
Speculative decoding is you have a smaller model, called a draft model, predict tokens ahead of the bigger model, and then you have the bigger model, verify all of this, all the tokens are predicted. And the reason it's faster is if you're predicting, one token at once, you're bound by memory bandwidth.
verbatim transcript · starts at 16:52
Vibhu [00:16:51]: I think it doesn’t hurt, right?
Akshat [00:16:52]: Yeah. Speculative decoding is you have a smaller model, called a draft model, predict tokens ahead of the bigger model, and then you have the bigger model, verify all of this, all the tokens are predicted. And the reason it’s faster is if you’re predicting, one token at once, you’re bound by memory bandwidth. But if you can batch the verification of, the draft model, then you’re much more efficient using compute, and it’s faster, and as long as your draft model is producing a lot of tokens that can get accepted, which is called the accept length, you can get a speed up that’s, multiple times of, the original model speed. and well, that’s what we highlight here. It’s Like people talk a lot about we made these kernels faster and whatnot, but improving kernel will only give you like few percentage points of improvement, and, increasing accept length, literally is a multiplicative decrease
Vibhu [00:17:47]: Like two to four X.
Akshat [00:17:48]: Yeah, exactly.