FactArticle

Qwen3.6, likely the most popular open-weight LLM series using a hybrid design, substitutes Gated DeltaNet layers for Mamba-2 in its non-attention portions.

Raschka notes the hybrid attention/state-space trend has spread to Qwen3.6, the most popular open-weight model line using this approach, though it swaps in Gated DeltaNet layers instead of Mamba-2. ✦ AI generated

Sebastian Raschka · Ahead of AI · 2026-06-06 · original ↗

Export .md Export .txt

This hybrid-architecture trend with alternating attention and alternative layers is a relatively popular development this year. The probably most popular open-weight LLM series that uses a similar hybrid design is probably Qwen3.6, which uses Gated DeltaNet layers instead of Mamba-2 layers for the non-attention portions.

Read full article ↗excerpt · fair-use quotation

✳ Gated DeltaNet ✳ Qwen3.6

Around this claim

Context · 1

This list is a curated, personally biased selection of papers I found interesting or relevant to my own work, not a complete record of everything published this year.Sebastian Raschka · Ahead of AI · conf 70%

This moment responds to

gives example → 2026 architecture research has moved beyond simply scaling transformers larger, branching into hybrid architectures, state space layers, MoE capacity allocation, activation behavior, and representation geometry.Sebastian Raschka · Ahead of AI