ContextArticle
The KV cache grows with every token, and in a long context it can take up significant GPU memory — for a 70B model at 128K tokens, roughly 40 gigabytes.
Explains that the KV cache, which stores key and value vectors for every token, is the main driver of memory cost in long-context LLM inference, growing linearly with token count. ✦ AI generated
Author · ByteByteGo Newsletter · 2026-08-04 · original ↗
A key part of the answer lies in a block of working memory called the KV cache. This memory is built up while the model generates a response. It is separate from the knowledge stored in the model's weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU. For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.
Read full article ↗excerpt · fair-use quotation
Around this claim
This moment responds to
provides context → Gemma 4's E2B and E4B models share KV cache tensors across layers, cutting KV cache memory by roughly half in long-context settings.Sebastian Raschka · Ahead of AIprovides context → Grouped-query attention and multi-head latent attention both shrink the cache footprint per token, but GQA is the safe default while latent attention saves more memory at the cost of serving complexity.Author · ByteByteGo Newsletterprovides context → Eviction techniques that drop tokens from the cache to save memory face a structural problem: whether a token matters depends on a question that hasn't arrived yet, so an aggressively trimmed cache can miss information needed later.Author · ByteByteGo Newsletter