Quantization halves the cache by moving from 16-bit to 8-bit storage, and halves it again at 4 bits, with quality costs that are negligible at 8 bits but become measurable on demanding tasks at 4 bits.

Describes how quantizing key-value vectors from 16-bit to 8-bit or 4-bit storage directly halves the cache size, with the quality trade-off becoming significant at 4 bits, especially on multi-needle retrieval tasks. ✦ AI generated

Author · ByteByteGo Newsletter · 2026-08-04 · original ↗

Quantization goes after the bytes per number. The keys and values are usually stored at 16 bits each, and quantization rounds them to a smaller format such as 8 bits or 4 bits. Since the bytes-per-number term sits right there in the equation, moving from 16 bits to 8 halves the whole cache, and going to 4 bits halves it again. The appeal is that this applies to a model we already have and skips retraining entirely. The quality cost depends on how far we push it. Eight-bit storage often costs well under a percent of accuracy, which puts it within the noise for most workloads. Four-bit storage saves more and starts to show measurable losses on demanding tasks such as multi-needle retrieval, where the model has to pull several specific facts out of a long context.

Read full article ↗excerpt · fair-use quotation