Mechanism◆Audio · 0:00 · 2m
If you don't batch together many concurrent users on a GPU cluster, the cost per token can be a thousand times worse than if you do batch them, because batching amortizes the fixed cost of fetching model weights from memory across many tokens at once.
Reiner Pope · Dwarkesh Podcast