The expense of long-context generation comes less from holding the cache and more from sweeping through all of it on every single decoding token — the cache is a bandwidth cost as much as a storage one.

The first phase is prefill, where the model reads the entire input at once. It processes all of the input tokens in parallel and builds their key and value vectors into the cache in a single pass. Prefill keeps the GPU's math units busy, so we call it compute-bound, meaning the limit is how fast the chip can do arithmetic. The second phase is decoding, where the model produces the output one token at a time. Each new token runs an attention step against the whole cache, which means the model reads every stored key and value out of GPU memory before it can emit the next token. It repeats that read for every token it produces. The limit here is how fast the cache can move from memory into the compute units, so we call decoding memory-bound. The expense of long-context generation comes less from holding the cache and more from sweeping through all of it on every single token. A larger cache means more data crossing the memory bus per token, which shows up directly as slower and costlier generation.

Read full article ↗excerpt · fair-use quotation