A hybrid architecture combining Mamba state-space layers with attention layers and mixture-of-experts makes models both fast and capable — Mamba provides efficient global context at linear cost, attention supplies pinpoint recall, and MoE gives large total capacity at low per-token cost.

NVIDIA hybridizes mostly-Mamba layers (linear cost, constant memory, billion-token context windows) with a few attention blocks (exact recall of buried facts) plus MoE (activates only a subset of parameters per token, enabling large models at low per-token cost). ✦ AI generated

Bryan Catanzaro · ByteByteGo Newsletter · 2026-07-27 · original ↗

A hybrid architecture swaps most of the attention layers in the Transformer with Mamba layers and keeps a few blocks with full attention. NVIDIA's models rely on the same hybrid pattern. Most of the layers are Mamba, which keeps long inputs cheap to process and is what makes a million-token context window practical rather than just theoretical. A few attention layers are placed in between to restore the pinpoint recall that Mamba gives up, so the model can still retrieve exact details from anywhere in the context. On top of the hybrid design, the models also use mixture-of-experts, or MoE, layers. An MoE layer consists of many small expert layers and routes each token to only a few of them. So instead of running every parameter on every token, the model activates only a small subset of its parameters. That allows us to use a large total capacity at a low per-token cost, which keeps the model fast.

Read full article ↗excerpt · fair-use quotation