ATRIUMsearch → argument graph
Mechanism◆Audio · 0:00 — 1:30

If you don't batch together many concurrent users on a GPU cluster, the cost per token can be a thousand times worse than if you do batch them, because batching amortizes the fixed cost of fetching model weights from memory across many tokens at once.

Reiner Pope explains that the dramatic price/speed tiers offered by 'Fast Mode' APIs come down to batch size: unbatched serving can be ~1000x more expensive per token than properly batched serving, because weight-fetch costs are fixed per forward pass regardless of batch size. ✦ AI generated

Reiner Pope · Dwarkesh Podcast · 2026-04-29 · original ↗

Export .mdExport .txt

plays this moment only · 0:00 — 1:30

Elicited by

“Why is it the case that you can pay more to get faster latency?”

What will turn out to be the case is that if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together. We'll be able to see that quite explicitly.

verbatim transcript · starts at 0:00

Transcript · around this moment

0:00– How batch size affects token cost and speed

0:00– How batch size affects token cost and speed

▦ MatX✳ Blackwell NVL72
Related moments