Article · 2026-05-16 · 6 moments

Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention

From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs ✦ AI generated

Data◆

DeepSeek V4-Pro's combined CSA/HCA compressed attention, which shortens the KV cache along the sequence dimension rather than per-token, cuts single-token inference FLOPs to 27% and KV cache size to 10% of DeepSeek V3.2 at a 1M-token context.

DeepSeek V4 alternates a milder sparse compression (CSA) with a much heavier dense compression (HCA) that shrink the KV cache by summarizing groups of tokens, delivering dramatic reported efficiency gains over DeepSeek V3.2 at million-token context lengths.

transcript

Sebastian Raschka: The DeepSeek V4 paper reports that, at a 1M-token context length, DeepSeek V4-Pro uses only 27% of the single-token inference FLOPs and 10% of the KV cache size compared with DeepSeek V3.2, which uses MLA and DeepSeek Sparse Attention (DSA).

Mechanism

DeepSeek V4's manifold-constrained hyper-connections (mHC) widen the residual stream into several parallel streams while forcing the mixing matrices to be doubly stochastic, making the wider residual pathway stable enough to scale to production with minimal overhead.

DeepSeek V4 productionizes manifold-constrained hyper-connections, replacing the single residual stream with several interacting streams whose mixing matrices are constrained to be non-negative and row/column-normalized, stabilizing the wider residual pathway at large scale.

transcript

Sebastian Raschka: In mHC, this residual mapping is projected onto the manifold of doubly stochastic matrices, meaning all entries are non-negative and each row and column sums to 1. This makes the residual mixing behave more like a stable redistribution of information across streams.

Mechanism

Gemma 4's per-layer embeddings give each transformer layer its own small token-specific vector, adding representational capacity without scaling the whole transformer stack to a larger parameter count.

PLE is a separate efficiency trick from KV-sharing: it stores extra per-layer, per-token embedding capacity that is cheap to look up, letting Gemma 4's 'E' models have more effective capacity than their 'active' parameter count suggests.

transcript

Sebastian Raschka: The PLE design keeps the expensive transformer blocks closer to the smaller "effective" size, while storing additional capacity in per-layer embedding tables. These are much cheaper to use than adding more attention or FFN weights, since they are mainly lookup-style parameters that can be cached.

Mechanism

Laguna XS.2 assigns different numbers of query heads to different layers, giving sliding-window layers more query heads and global/full-attention layers fewer, while keeping the KV heads fixed at 8.

Poolside's Laguna XS.2 introduces per-layer query-head budgeting, spending more query heads on cheap sliding-window layers and fewer on expensive global-attention layers, rather than giving every layer the same attention budget.

transcript

Sebastian Raschka: So Laguna XS.2 gives more query heads to sliding-window layers and fewer query heads to global layers, while keeping the KV heads fixed at 8. That is the actual layer-wise head budgeting in the config.

Mechanism

ZAYA1-8B's Compressed Convolutional Attention performs the attention operation directly inside a compressed latent space, rather than compressing only the stored KV cache as MLA does, which reduces both KV cache size and attention FLOPs.

Zyphra's ZAYA1-8B uses Compressed Convolutional Attention, which compresses Q, K, and V and computes attention directly in that compressed space, with added convolutional mixing on Q/K to preserve local context that compression would otherwise lose.

transcript

Sebastian Raschka: CCA compresses Q, K, and V and performs the attention operation directly in the compressed latent space. This is why CCA can reduce not only KV cache size, but also attention FLOPs during prefill and training.

Highlight slides

Gemma 4 Small Models Halve KV Cache Memory✦ from: Gemma 4's E2B and E4B models share KV cache tensors across layers, cutting KV cache memory by roughly half in long-context settings.DeepSeek V4-Pro: Compressed Attention Gains✦ from: DeepSeek V4-Pro's combined CSA/HCA compressed attention, which shortens the KV cache along the sequence dimension rather than per-token, cuts single-token inference FLOPs to 27% and KV cache size to 10% of DeepSeek V3.2 at a 1M-token context.Sebastian Raschka: How the Savings Work✦ from: Gemma 4's E2B and E4B models share KV cache tensors across layers, cutting KV cache memory by roughly half in long-context settings.How CSA/HCA Compression Works✦ from: DeepSeek V4-Pro's combined CSA/HCA compressed attention, which shortens the KV cache along the sequence dimension rather than per-token, cuts single-token inference FLOPs to 27% and KV cache size to 10% of DeepSeek V3.2 at a 1M-token context.

Related episodes

Article · Ahead of AI

LLM Research Papers: The 2026 List (January to May)

→