Grouped-query attention and multi-head latent attention both shrink the cache footprint per token, but GQA is the safe default while latent attention saves more memory at the cost of serving complexity.

Describes two architectural approaches: grouped-query attention (sharing key-value heads across query heads) and multi-head latent attention (compressing keys/values into a latent space), noting their trade-offs in memory savings, quality, and serving complexity. ✦ AI generated

Author · ByteByteGo Newsletter · 2026-08-04 · original ↗

Grouped-query attention goes after the key-value head count. In a standard attention layer, every query head carries its own key and value head, so a model with 64 query heads stores 64 sets of keys and values. Grouped-query attention lets several query heads share one key-value head, which drops the number of stored sets sharply. For example, Llama 2 and 3 at 70B and Mistral 7B share down to 8 key-value heads, which cuts the cache by roughly eightfold against full multi-head attention. This is why a recent 70B model can hold a smaller cache than an older 7B one. Multi-head latent attention, introduced in the DeepSeek models, projects the keys and values down into a smaller latent representation before caching them, then expands them back when they are read. The savings are large. DeepSeek-V3 holds around 70 kilobytes per token, where comparable grouped-query models sit between 192 and 328. The cost lies in serving, since the compression adds work on every read and pairs awkwardly with some standard attention implementations, so it tends to pay off most once models and contexts grow large enough that cache traffic dominates.

Read full article ↗excerpt · fair-use quotation