Eviction techniques that drop tokens from the cache to save memory face a structural problem: whether a token matters depends on a question that hasn't arrived yet, so an aggressively trimmed cache can miss information needed later.

Explains that eviction approaches (like keeping a recent window plus attention sinks) face a fundamental problem: you cannot know which tokens future generation steps will need, so aggressive eviction causes retrieval failures on long documents. ✦ AI generated

Author · ByteByteGo Newsletter · 2026-08-04 · original ↗

Eviction goes after the token count by dropping entries that the model is unlikely to need. The common approach keeps a window of the most recent tokens, since recent context usually matters most, along with a few tokens from the very start of the sequence. Those opening tokens turn out to play an outsized role. They absorb a large share of attention regardless of what they actually say, acting as anchors that keep the model's output stable. The trouble with eviction is structural. Whether a token matters depends on a question that has yet to arrive. A token we drop now can be exactly the one a later part of the generation needs, and once it is gone, the model generates as though that token had been absent the whole time. This shows up on retrieval tasks, where an aggressively trimmed cache handles a casual chat well and then misses a fact buried in the middle of a long document.

Read full article ↗excerpt · fair-use quotation