Why An LLM’s Memory Gets Expensive and How to Fix It

Author: Eviction goes after the token count by dropping entries that the model is unlikely to need. The common approach keeps a window of the most recent tokens, since recent context usually matters most, along with a few tokens from the very start of the sequence. Those opening tokens turn out to play an outsized role. They absorb a large share of attention regardless of what they actually say, acting as anchors that keep the model's output stable. The trouble with eviction is structural. Whether a token matters depends on a question that has yet to arrive. A token we drop now can be exactly the one a later part of the generation needs, and once it is gone, the model generates as though that token had been absent the whole time. This shows up on retrieval tasks, where an aggressively trimmed cache handles a casual chat well and then misses a fact buried in the middle of a long document.

Author: Grouped-query attention goes after the key-value head count. In a standard attention layer, every query head carries its own key and value head, so a model with 64 query heads stores 64 sets of keys and values. Grouped-query attention lets several query heads share one key-value head, which drops the number of stored sets sharply. For example, Llama 2 and 3 at 70B and Mistral 7B share down to 8 key-value heads, which cuts the cache by roughly eightfold against full multi-head attention. This is why a recent 70B model can hold a smaller cache than an older 7B one. Multi-head latent attention, introduced in the DeepSeek models, projects the keys and values down into a smaller latent representation before caching them, then expands them back when they are read. The savings are large. DeepSeek-V3 holds around 70 kilobytes per token, where comparable grouped-query models sit between 192 and 328. The cost lies in serving, since the compression adds work on every read and pairs awkwardly with some standard attention implementations, so it tends to pay off most once models and contexts grow large enough that cache traffic dominates.

Author: Quantization goes after the bytes per number. The keys and values are usually stored at 16 bits each, and quantization rounds them to a smaller format such as 8 bits or 4 bits. Since the bytes-per-number term sits right there in the equation, moving from 16 bits to 8 halves the whole cache, and going to 4 bits halves it again. The appeal is that this applies to a model we already have and skips retraining entirely. The quality cost depends on how far we push it. Eight-bit storage often costs well under a percent of accuracy, which puts it within the noise for most workloads. Four-bit storage saves more and starts to show measurable losses on demanding tasks such as multi-needle retrieval, where the model has to pull several specific facts out of a long context.

Author: A key part of the answer lies in a block of working memory called the KV cache. This memory is built up while the model generates a response. It is separate from the knowledge stored in the model's weights, and it holds the key and value vectors computed for every token of the input. The cache grows with every token, and in a long context, it can take up significant space on the GPU. For example, for a 70-billion-parameter model at a context of 128,000 tokens, it comes to roughly 40 gigabytes, a serious amount of GPU memory that grows with every user you add.