ClaimAudio · 93:02 · 2m
LLM context lengths jumped from ~8K to ~100-200K tokens between GPT-3 and GPT-4, then plateaued there for the last year or two, indicating that this range represents a cost-balanced equilibrium point set by memory bandwidth, beyond which scaling further becomes cost-prohibitive.
Reiner Pope · Dwarkesh Podcast