Data◆Article

DeepSeek V4-Pro's combined CSA/HCA compressed attention, which shortens the KV cache along the sequence dimension rather than per-token, cuts single-token inference FLOPs to 27% and KV cache size to 10% of DeepSeek V3.2 at a 1M-token context.

DeepSeek V4 alternates a milder sparse compression (CSA) with a much heavier dense compression (HCA) that shrink the KV cache by summarizing groups of tokens, delivering dramatic reported efficiency gains over DeepSeek V3.2 at million-token context lengths. ✦ AI generated

Sebastian Raschka · Ahead of AI · 2026-05-16 · original ↗

Export .md Export .txt

The DeepSeek V4 paper reports that, at a 1M-token context length, DeepSeek V4-Pro uses only 27% of the single-token inference FLOPs and 10% of the KV cache size compared with DeepSeek V3.2, which uses MLA and DeepSeek Sparse Attention (DSA).

Read full article ↗excerpt · fair-use quotation

✳ CSA ✳ DeepSeek V3.2 ✳ DeepSeek V4 ✳ HCA

Related moments

Mechanism◆Article

Nemotron 3 alternates regular attention layers with Mamba-2 state-space layers to improve long-context efficiency, which has become the central priority in 2026 as more LLMs get embedded in agent harnesses requiring longer contexts.

Sebastian Raschka · Ahead of AI

Mechanism◆Article

Gemma 4's E2B and E4B models share KV cache tensors across layers, cutting KV cache memory by roughly half in long-context settings.

Sebastian Raschka · Ahead of AI

PredictionArticle

Compared to his 2025 lists, this year's bookmarks skew more toward agent harnesses, tool use, long context, diffusion language models, and practical serving infrastructure, reflecting where the field is heading.

Sebastian Raschka · Ahead of AI