Mechanism◆Audio · 0:00 — 1:30
If you don't batch together many concurrent users on a GPU cluster, the cost per token can be a thousand times worse than if you do batch them, because batching amortizes the fixed cost of fetching model weights from memory across many tokens at once.
Reiner Pope explains that the dramatic price/speed tiers offered by 'Fast Mode' APIs come down to batch size: unbatched serving can be ~1000x more expensive per token than properly batched serving, because weight-fetch costs are fixed per forward pass regardless of batch size. ✦ AI generated
Reiner Pope · Dwarkesh Podcast · 2026-04-29 · original ↗
plays this moment only · 0:00 — 1:30
Elicited by
“Why is it the case that you can pay more to get faster latency?”
What will turn out to be the case is that if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together. We'll be able to see that quite explicitly.
verbatim transcript · starts at 0:00
Transcript · around this moment
0:00– How batch size affects token cost and speed
0:00– How batch size affects token cost and speed
- ·Unbatched GPU serving can cost 1000x more per token
- ·Batching many users together fixes this gap
- ·Effect explains tiered 'Fast Mode' API pricing
- ·Reiner Pope: effect will be 'quite explicit'
- ·Fetching model weights from memory is a fixed cost
- ·One user still pays that full fetch cost alone
- ·Batching amortizes it across many tokens at once
Around this claim