Audio · 2026-04-29 · 6 moments

Reiner Pope – The math behind how LLMs are trained and served

It's shocking how much you can deduce about what the labs are doing from a handful of equations and a blackboard ✦ AI generated

Apple Podcasts ↗

GUEST Reiner Pope HOST Dwarkesh Patel

timeline · colored by role

Mechanism◆

If you don't batch together many concurrent users on a GPU cluster, the cost per token can be a thousand times worse than if you do batch them, because batching amortizes the fixed cost of fetching model weights from memory across many tokens at once.

Reiner Pope explains that the dramatic price/speed tiers offered by 'Fast Mode' APIs come down to batch size: unbatched serving can be ~1000x more expensive per token than properly batched serving, because weight-fetch costs are fixed per forward pass regardless of batch size.

transcript

Reiner Pope: What will turn out to be the case is that if you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together. We'll be able to see that quite explicitly.

Mechanism

A mixture-of-experts layer's all-to-all communication pattern fits perfectly within a single GPU rack's fully-connected scale-up network, but breaks down across rack boundaries, so the physical size of a rack sets a hard ceiling on how large (how sparse) an expert layer can practically be.

Because every GPU may need to send tokens to every other GPU in an MoE layer, and rack-to-rack links are much slower than within-rack links, Reiner Pope argues the rack has become the natural unit that bounds expert-parallel layer size, driving the industry push toward ever-larger interconnect domains.

transcript

Reiner Pope: The fundamental thing here is that one rack bounds the size of an expert layer you can do. This has been part of what's been driving towards larger and larger interconnect domains.

Mechanism

Pipeline parallelism during inference doesn't reduce compute time or memory-fetch time at all — it just relocates the memory fetch from one chip to another — so its only real benefit is reducing per-GPU memory capacity requirements, not latency or throughput.

Responding to Dwarkesh's mention of Ilya Sutskever's remark that 'pipelining is not wise,' Reiner Pope clarifies that pipeline parallelism in inference saves memory capacity only, not runtime, and that this benefit further cancels out for the KV cache because sharding across pipeline stages requires proportionally more sequences in flight.

transcript

Reiner Pope: In inference, what are we saving on? Are we saving on memory time or compute time? Not really. We're just moving the memory time from one chip to another chip, or one rack to a different rack. There's no actual benefit in runtime. However, what we are saving on is memory capacity.

Data◆

Back-of-envelope math comparing global inference token volume to rumored pre-training token counts (~150 trillion) versus the Chinchilla-optimal token count for a ~100B-active-parameter model (~2 trillion) implies frontier models are being trained on roughly 100x more data than Chinchilla scaling laws would recommend.

By assuming training and inference compute costs should roughly equalize, and plugging in estimates of global tokens-per-second and model active-parameter counts, Reiner Pope derives that frontier models are likely over-trained by about a factor of 100 relative to the Chinchilla-optimal point — a consequence of needing cheap inference and RL-generation compute, not just training-optimal quality.

transcript

Reiner Pope: The ratio of this two hundred trillion or a hundred trillion parameters over the Chinchilla optimal of two trillion, that's the amount it's over-trained. Which is a factor of a hundred over-trained.

Claim

LLM context lengths jumped from ~8K to ~100-200K tokens between GPT-3 and GPT-4, then plateaued there for the last year or two, indicating that this range represents a cost-balanced equilibrium point set by memory bandwidth, beyond which scaling further becomes cost-prohibitive.

Arguing from API pricing structures and roofline cost curves, Reiner Pope contends that context lengths stalled around 100-200K tokens because memory bandwidth (not compute, and not fully solved by sparse attention) is the fundamental bottleneck, casting doubt on paths to 100-million-token contexts without new hardware.

transcript

Reiner Pope: But if you look at the history of context lengths of models, from earlier models like GPT-3, maybe to GPT-4—I don't remember when the transition happened exactly—they shot up from about 8K to 100-200K. And then for the last year or two, they've all been hovering around there. I think that indicates that this is the reasonably balanced cost point, and going massively beyond that would be cost-prohibitive.

Mechanism

Neural networks and cryptographic ciphers both rely on scrambling/mixing information across inputs, but they diverge sharply in their optimization goals: ciphers are built to make small input differences produce huge, unpredictable output differences (the avalanche property, exploited by differential cryptanalysis), whereas neural nets use residual connections and LayerNorm specifically to keep gradients smooth and differentiable rather than chaotic.

Reiner Pope contrasts neural nets and ciphers as convergent-but-opposite designs: both scramble information, but ciphers are optimized to blow up small input differences (differential cryptanalysis targets this), while neural nets use residual connections and LayerNorm to keep the system smoothly differentiable for gradient descent.

transcript

Reiner Pope: Basically, what it says is that if you take a small difference of the input, it's quite difficult to make the difference of the output be small. The whole job of a well-designed cipher is to make the difference in output very large. The distinction is that the optimization goals at that point are about complexifying. They don't have the same residual connections, like LayerNorms.

Highlight slides

Batching Decides the Cost of a Token✦ from: If you don't batch together many concurrent users on a GPU cluster, the cost per token can be a thousand times worse than if you do batch them, because batching amortizes the fixed cost of fetching model weights from memory across many tokens at once.Why: Fixed Cost, Shared Across Users✦ from: If you don't batch together many concurrent users on a GPU cluster, the cost per token can be a thousand times worse than if you do batch them, because batching amortizes the fixed cost of fetching model weights from memory across many tokens at once.Frontier Models Are Over-Trained by Design✦ from: Back-of-envelope math comparing global inference token volume to rumored pre-training token counts (~150 trillion) versus the Chinchilla-optimal token count for a ~100B-active-parameter model (~2 trillion) implies frontier models are being trained on roughly 100x more data than Chinchilla scaling laws would recommend.The Token Gap vs. Chinchilla-Optimal✦ from: Back-of-envelope math comparing global inference token volume to rumored pre-training token counts (~150 trillion) versus the Chinchilla-optimal token count for a ~100B-active-parameter model (~2 trillion) implies frontier models are being trained on roughly 100x more data than Chinchilla scaling laws would recommend.Why Over-Train Past Chinchilla?✦ from: Back-of-envelope math comparing global inference token volume to rumored pre-training token counts (~150 trillion) versus the Chinchilla-optimal token count for a ~100B-active-parameter model (~2 trillion) implies frontier models are being trained on roughly 100x more data than Chinchilla scaling laws would recommend.

Related episodes

Video · Machine Learning Street Talk

If you don't batch together many concurrent users on a GPU cluster, the cost per token can be a thousand times worse than if you do batch them, because batching amortizes the fixed cost of fetching model weights from memory across many tokens at once.

A mixture-of-experts layer's all-to-all communication pattern fits perfectly within a single GPU rack's fully-connected scale-up network, but breaks down across rack boundaries, so the physical size of a rack sets a hard ceiling on how large (how sparse) an expert layer can practically be.

Pipeline parallelism during inference doesn't reduce compute time or memory-fetch time at all — it just relocates the memory fetch from one chip to another — so its only real benefit is reducing per-GPU memory capacity requirements, not latency or throughput.

LLM context lengths jumped from ~8K to ~100-200K tokens between GPT-3 and GPT-4, then plateaued there for the last year or two, indicating that this range represents a cost-balanced equilibrium point set by memory bandwidth, beyond which scaling further becomes cost-prohibitive.

The Thermodynamic AI Chip · Thomas Ahle

sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

Using Local Coding Agents