Atrium — Attention is all that matters

24 moments across 7 channels for “Architectural structure/bias (gating, nested nonlinearity) impro”

Core claim

Claim◆34:08

The optimal amount of architectural bias (gating, nonlinearity, structure) in a neural network is inversely related to model scale: smaller, specialized models benefit from more biased/structured operators, while at very large scale (up to trillions of parameters) unstructured, unbiased architectures like pure attention and matrix multiplication win.

Hassani explains Liquid's key architectural finding: bias-heavy, structured architectures (gating, nonlinearity) win at small scale, but as models grow toward trillions of parameters, unstructured operators like pure attention and matrix multiplication dominate.

Ramin Hasani · The Cognitive Revolution · listen to the original →

4 restatements of this claim folded — Ramin Hasani, Ramin Hasani, Ramin Hasani, Ramin Hasani

In practice · 1

With just 12 liquid neurons a system could autonomously parallel park a car; 19 neurons could drive a car; 30 neurons could autonomously fly a drone.

Ramin Hasani · The Cognitive Revolution · 12:45conf 60%

Counterpoint · 2

At smaller, specialized model scales, biased/structured components like gating mechanisms improve performance, but as models scale up they benefit from becoming maximally unstructured (pure attention, pure matrix multiplication)—architectural bias is a function of scale, not a universal good.

Ramin Hassani · The Cognitive Revolution · 63:27conf 70%

As neural networks scale up in size, the optimal design becomes progressively less structured, which is why unbiased architectures like transformers dominate at the largest scales.

Ramin Hasani · The Cognitive Revolution · 33:43conf 70%

More signal

Mechanism◆Video · 42:17 · 2m

Liquid AI built an in-house automated architecture search system (AFMD) that puts target hardware in the loop and evaluates candidate architectures on real downstream application performance across ~100 benchmarks rather than proxy metrics like perplexity, specifically to remove human bias from architecture design.

Hassani describes AFMD, Liquid's automated foundation model design system, which uses an evolution strategy with real hardware and downstream-task evaluation to eliminate human bias from architecture choices — a deliberate reaction against how architecture decisions are typically made ad hoc by a small group of experts at frontier labs.

Ramin Hasani · The Cognitive Revolution

Mechanism◆Audio · 14:08 · 5m

Every car on the road except Tesla and Rivian uses a domain-based architecture with 100-150 separate ECUs, which is a disastrous legacy that makes software updates nearly impossible — the reason Volkswagen paid Rivian $5.8 billion to license Rivian's zonal architecture.

Scaringe contrasts the traditional domain-based architecture (100-150 separate computers, each running supplier-written software on its own island) with Rivian's zonal architecture (1-3 computers running a single OS). He cites Rivian's $5.8 billion deal with Volkswagen Group as evidence that legacy automakers recognize they cannot build this capability internally.

RJ Scaringe · No Priors

ContextAudio · 6:18 · 1m

New compute workloads create windows of opportunity for new architectures — just as graphics created Nvidia and mobile created ARM, AI's emergence as a new workload demanded a dedicated, non-derivative architecture.

Feldman explains the strategic thesis behind Cerebras: every major new compute workload — graphics, mobile — created a new dominant architecture while incumbents like Intel and AMD got zero share. AI represented the same kind of opportunity, demanding a clean-sheet design, not a derivative of existing architectures.

Andrew Feldman · No Priors

MechanismVideo · 43:09 · 2m

Liquid AI built an automated, hardware-in-the-loop architecture search system (AFMD) specifically to eliminate human bias in architecture design, because even at top labs a small group of people is really just calling the shots based on personal intuition.

Hassani explains that Liquid built its AFMD automated architecture-search system with hardware-in-the-loop evolutionary optimization specifically to remove the human biases he says still drive architecture decisions even at top labs like Anthropic and OpenAI.

Ramin Hasani · The Cognitive Revolution

Claim◆Video · 12:12 · 2m

The single most important property of a computing platform is its install base, not the elegance of its architecture.

Jensen argues that a computing architecture's install base is the decisive factor for its success, using x86's triumph over more elegant RISC architectures as evidence.

Jensen Huang · Lex Fridman

ClaimArticle · 53 words

2026 architecture research has moved beyond simply scaling transformers larger, branching into hybrid architectures, state space layers, MoE capacity allocation, activation behavior, and representation geometry.

Raschka observes that 2026 architecture innovation spans several new directions—hybrid designs, state space layers, MoE capacity routing, and more—rather than just building bigger transformers.

Sebastian Raschka · Ahead of AI

MechanismAudio · 3:15 · 1m

Wafer-scale engineering was the right architectural bet because radical improvement requires radical architectural difference, not incremental modification of the GPU.

Feldman explains Cerebras's bet on wafer-scale chips: to be 15-20x faster than GPUs, you cannot make minor modifications — you need a fundamentally different architecture, which is why Cerebras built a dinner-plate-sized chip.

Andrew Feldman · No Priors

ClaimVideo · 53:41 · 2m

A year and a half before Mamba, our Liquid-S4 paper first introduced input-dependent SSMs — the gating mechanism that later became central to Mamba and other sequence architectures originated in our liquid neural network research.

Hasani traces the input-dependent gating mechanism now central to architectures like Mamba back to Liquid AI's own Liquid-S4 paper, published a year and a half earlier, framing it as a foundational discovery rooted in liquid neural network theory.

Ramin Hasani · The Cognitive Revolution

DataVideo · 14:51 · 2m

The learned 'principal odor map' embedding organizes molecules so that perceptually related scents form nested sub-clusters (e.g., jasmine, rose, and violet nested inside a broader floral region) without ever being explicitly told this hierarchical structure exists.

When Osmo visualized their learned odor embedding space, semantically related scent categories clustered into nested neighborhoods (florals containing jasmine/rose/violet) that the model was never told to form.

Alex Wiltschko · The TWIML AI Podcast

AnecdoteVideo · 4:42 · 3m

Anthropic found its product identity not through a grand strategy, but through small, bottoms-up experiments like Golden Gate Claude that showcased research in an authentic, differentiated way.

Dianne describes how shipping Golden Gate Claude — a quirky 24-hour experiment where Claude obsessed over the Golden Gate Bridge — was an early inflection point that showed Anthropic could build products different from competitors.

Dianne Penn · Lenny's Podcast

Claim◆Video · 16:27 · 16m

Chomsky's poverty of stimulus argument, that it is impossible to learn to become creative from examples alone, is refuted: deep architectures have a huge implicit bias to build coarse-grained hierarchical variables, so they can learn to be creative from polynomially many (not exponentially many) sentences.

In Wyart's synthetic tree-structured world, a shallow network does exactly what Chomsky predicted — it memorizes and cannot generalize. But deep architectures exhibit a strong implicit bias to construct coarse-grained hierarchical variables, learning to be creative from only polynomially many sentences. This is a counterexample showing that what must be 'innate' shrinks dramatically for a deep architecture.

Matthieu Wyart · Machine Learning Street Talk

MechanismVideo · 29:50 · 2m

State space models rely on linear dynamics because current methods cannot properly scale nonlinear systems into parallelizable tensor computation.

Hasani explains that the core bottleneck in scaling architectures like liquid neural networks is that nonlinear relationships resist being cleanly tensorized for parallel computation, which is why state space models default to linear dynamics.

Ramin Hasani · The Cognitive Revolution

ClaimAudio · 24:57 · 3m

Large language models are not made for predictive analytics; for forecasting demand, cash flow, and classification-regression questions, classical machine learning approaches like XGBoost or AutoML are still necessary, and SAP has developed a new transformer architecture called RPT (Relational Pre-trained Transformers) to democratize predictions over structured data.

Philipp Herzig explains that while LLMs excel at unstructured text and images, enterprise planning requires predictions — demand forecasting, cash flow prediction, classification of customer payment behavior, regression of payment delays. LLMs are not designed for these tasks because they generate one token at a time in sequence-to-sequence modeling. Classical ML approaches like XGBoost work but don't scale because they require hiring data scientists and training separate models per country. SAP's RPT (Relational Pre-trained Transformer) applies the transformer architecture to structured tabular data, enabling accurate predictions with small amounts of data.

Philipp Herzig · No Priors