LLM context lengths jumped from ~8K to ~100-200K tokens between GPT-3 and GPT-4, then plateaued there for the last year or two, indicating that this range represents a cost-balanced equilibrium point set by memory bandwidth, beyond which scaling further becomes cost-prohibitive.

Arguing from API pricing structures and roofline cost curves, Reiner Pope contends that context lengths stalled around 100-200K tokens because memory bandwidth (not compute, and not fully solved by sparse attention) is the fundamental bottleneck, casting doubt on paths to 100-million-token contexts without new hardware. ✦ AI generated

Reiner Pope · Dwarkesh Podcast · 2026-04-29 · original ↗

But if you look at the history of context lengths of models, from earlier models like GPT-3, maybe to GPT-4—I don't remember when the transition happened exactly—they shot up from about 8K to 100-200K. And then for the last year or two, they've all been hovering around there. I think that indicates that this is the reasonably balanced cost point, and going massively beyond that would be cost-prohibitive.

verbatim transcript · starts at 93:02