Paged attention and prefix caching eliminate memory fragmentation and enable cache sharing across requests, cutting wasted memory from 60-80% to under 4% and reducing latency by 50-90% on cache hits.

Older serving systems reserved one large contiguous block per request, sized for the longest output it might produce. Most requests finished well short of that, leaving the reserved space idle, and the fragmentation added up. Paged attention borrows an idea from operating systems, which break memory into small fixed-size pages and hand them out on demand. The cache gets split into small blocks that can live anywhere in memory, tracked by a lookup table that maps each request to its blocks. The result is that systems that wasted 60 to 80 percent of cache memory to fragmentation dropped that figure below 4 percent, and throughput climbed by two to three times, all from packing the same data more tightly. Since the cache lives in shareable blocks, two requests that begin with the same text can point at the same physical blocks while each holds its own private continuation. This is the foundation of prefix caching, and the productized version that the major APIs call prompt caching. The win is large for any workload that repeats a prefix, such as an agent that sends the same multi-thousand-token system prompt on every call. OpenAI and Anthropic both report cost and latency reductions of 50 to 90 percent on cache hits, with cached tokens billed at a fraction of fresh ones.

Read full article ↗excerpt · fair-use quotation