Data◆Audio · 78:59 — 80:29

Back-of-envelope math comparing global inference token volume to rumored pre-training token counts (~150 trillion) versus the Chinchilla-optimal token count for a ~100B-active-parameter model (~2 trillion) implies frontier models are being trained on roughly 100x more data than Chinchilla scaling laws would recommend.

By assuming training and inference compute costs should roughly equalize, and plugging in estimates of global tokens-per-second and model active-parameter counts, Reiner Pope derives that frontier models are likely over-trained by about a factor of 100 relative to the Chinchilla-optimal point — a consequence of needing cheap inference and RL-generation compute, not just training-optimal quality. ✦ AI generated

Reiner Pope · Dwarkesh Podcast · 2026-04-29 · original ↗

Export .md Export .txt

plays this moment only · 78:59 — 80:29

Elicited by

“Can we back out how much more compute than Chinchilla optimal for a given sized model?”

The ratio of this two hundred trillion or a hundred trillion parameters over the Chinchilla optimal of two trillion, that's the amount it's over-trained. Which is a factor of a hundred over-trained.

verbatim transcript · starts at 78:59

Transcript · around this moment

78:59– Because of RL, models may be 100x over-trained beyond Chinchilla-optimal

78:59– Because of RL, models may be over-trained 100x beyond Chinchilla-optimal

✳ Chinchilla scaling law ✳ GPT-5

Related moments

Mechanism◆Article

Newer Anthropic models have been specifically trained, presumably via reinforcement learning, to better use the edit tools baked into Claude Code, and this makes them more likely to misuse the differently-shaped custom edit tools of other coding harnesses like Pi.

Armin Ronacher · Simon Willison's Weblog

Mechanism◆Audio · 60:33 · 2m

MCTS-based training works because it never asks the network to directly chase a win/loss signal — instead, for every single action taken, search produces a strictly better relabeled target, giving one clean supervision signal per action instead of a noisy win-rate credit-assignment problem.

Eric Jang · Dwarkesh Podcast

MechanismAudio · 132:02 · 2m

RL's inefficiency is best understood as bits-per-FLOP, which decomposes into samples-per-FLOP (which falls as trajectories get longer) times bits-per-sample (which is far lower for RL's win/loss signal than for a supervised learning label).

Dwarkesh Patel · Dwarkesh Podcast