Atrium — Attention is all that matters

28 moments across 7 channels for “A closed-form solution to liquid neural network (continuous-time”

Core claim

Claim◆2:49

Trained neural network weights can be treated as an input modality for training another neural network to analyze and generate new weights, analogous to how language models learn from text and image models learn from pixels.

Borth introduces weight-space learning, arguing that trained model weights are not just the output of training but can be used as input data for a new class of neural networks that analyze and generate weights, just as language models process text or vision models process pixels.

Damian Borth · The TWIML AI Podcast · listen to the original →

Mechanism · 1

A neural network model is literally just a file containing weight values — numbers that were calculated through backpropagation training — and every neural network works this way, from a tiny XOR network to GPT.

CJ · Syntax · 18:31conf 70%

Evidence · 1

An autoencoder trained on a homogeneous population of neural networks can predict the accuracy, generalization gap, and activation function of unseen models from their weights alone, without requiring test data.

Damian Borth · The TWIML AI Podcast · 5:44conf 85%

In practice · 3

An autoencoder trained on a homogeneous population of neural networks can predict the accuracy, generalization gap, and activation function of unseen models from their weights alone, without requiring test data.

Damian Borth · The TWIML AI Podcast · 5:44conf 85%

Weight-space learning enables knowledge transfer across domains that generates domain-specific models with 20-30x less compute than training from scratch, demonstrated by generating remote sensing foundation models from an ImageNet-based encoder-decoder.

Damian Borth · The TWIML AI Podcast · 28:25conf 80%

Using dataset embeddings as prompts instead of model prompts enables privacy-preserving generation of neural network weights from sensitive data that cannot be shared, opening model generation to regulated industries.

Damian Borth · The TWIML AI Podcast · 32:08conf 80%

Context · 2

A neural network model is literally just a file containing weight values — numbers that were calculated through backpropagation training — and every neural network works this way, from a tiny XOR network to GPT.

CJ · Syntax · 18:31conf 85%

Mean squared error loss is fundamentally inadequate for generating functional neural network weights from a latent representation, because it reconstructs average weights well but loses the high-frequency information critical for network function.

Damian Borth · The TWIML AI Podcast · 10:52conf 60%

Extends · 3

Weight-space learning can be scaled beyond homogeneous model zoos to train on diverse open-weight models from Hugging Face across different architectures, tasks, and modalities.

Damian Borth · The TWIML AI Podcast · 20:54conf 85%

Weight-space learning enables knowledge transfer across domains that generates domain-specific models with 20-30x less compute than training from scratch, demonstrated by generating remote sensing foundation models from an ImageNet-based encoder-decoder.

Damian Borth · The TWIML AI Podcast · 28:25conf 80%

Using dataset embeddings as prompts instead of model prompts enables privacy-preserving generation of neural network weights from sensitive data that cannot be shared, opening model generation to regulated industries.

Damian Borth · The TWIML AI Podcast · 32:08conf 80%

Counterpoint · 1

Mean squared error loss is fundamentally inadequate for generating functional neural network weights from a latent representation, because it reconstructs average weights well but loses the high-frequency information critical for network function.

Damian Borth · The TWIML AI Podcast · 10:52conf 80%

More signal

Mechanism◆Video · 17:58 · 2m

We solved the closed-form solution for liquid neural network neuron dynamics in 2022, a problem that had gone unsolved since 1907, and this removed the need for numerical solvers so the networks can scale to billions of neurons.

Hassani describes how Liquid AI derived, for the first time in 2022, a closed-form solution to the differential equations governing liquid neural network dynamics — a problem open since 1907 — enabling scaling from hundreds of neurons to billions.

Ramin Hasani · The Cognitive Revolution

ClaimVideo · 53:41 · 2m

A year and a half before Mamba, our Liquid-S4 paper first introduced input-dependent SSMs — the gating mechanism that later became central to Mamba and other sequence architectures originated in our liquid neural network research.

Hasani traces the input-dependent gating mechanism now central to architectures like Mamba back to Liquid AI's own Liquid-S4 paper, published a year and a half earlier, framing it as a foundational discovery rooted in liquid neural network theory.

Ramin Hasani · The Cognitive Revolution

MechanismVideo · 28:02 · 2m

Nonlinear neural network dynamics cannot be trivially converted into parallelizable tensor computations, which is the fundamental scaling bottleneck for liquid neural networks and the reason state space models were built on linear dynamics instead.

Hassani explains that the core technical bottleneck in scaling liquid neural networks is that nonlinear relationships resist being reshaped into parallel tensor/matrix computations, which is why alternative architectures like state space models chose linear dynamics.

Ramin Hassani · The Cognitive Revolution

AnecdoteVideo · 12:45 · 2m

With as few as 12 to 30 liquid neurons, we demonstrated a car autonomously parallel parking, driving, and a drone flying — control tasks normally requiring vastly larger neural networks.

Ramin Hasani recounts early MIT results showing that tiny liquid neural networks — as few as 12 neurons — could autonomously park, drive, or fly a drone, far outperforming what network size would suggest.

Ramin Hasani · The Cognitive Revolution

ContextVideo · 9:16 · 2m

The C. elegans worm's 300-neuron nervous system achieves control and dexterity that outperforms the best robotic systems, which is what inspired Liquid AI's approach of modeling neural networks directly on biological neuron dynamics.

Ramin Hasani explains that Liquid AI's origins trace to studying the C. elegans worm, whose 300-neuron nervous system outperforms robotic control systems, motivating a biologically-grounded approach to neural network design.

Ramin Hasani · The Cognitive Revolution

ExampleVideo · 12:45 · 2m

Extremely small liquid neural networks—as few as 12, 19, and 30 neurons—were sufficient to autonomously parallel park a car, drive a car, and fly a drone, respectively.

Ramin Hassani recounts early MIT results showing tiny liquid neural networks with only a dozen or so neurons could handle real-world control tasks like parking and flying, far outperforming expectations for such small models.

Ramin Hasani · The Cognitive Revolution

Claim◆Video · 33:15 · 2m

The larger a neural network gets, the less structured and less biased it should be — transformers dominate at trillion-parameter scale, while gated, biased architectures win at smaller, specialized scales.

Hasani argues there's a 'scale-to-bias' law: unstructured architectures like pure transformers win at massive scale, while smaller, resource-constrained models benefit from biased, gated, structured operators like those in liquid networks.

Ramin Hasani · The Cognitive Revolution

MechanismVideo · 13:58 · 2m

The key structural difference between an energy-based model and a traditional feedforward neural network is where the cost function is applied: ordinary networks optimize a cost that's only a function of inputs and outputs, while energy-based models also apply a cost to internal latent states, requiring two separate minimizations.

Beck explains that energy-based models differ from standard neural nets because their cost function also constrains internal/latent states, not just the input-output mapping, requiring a dual minimization process.

Jeff Beck · Machine Learning Street Talk

MechanismAudio · 124:02 · 2m

Neural networks and cryptographic ciphers both rely on scrambling/mixing information across inputs, but they diverge sharply in their optimization goals: ciphers are built to make small input differences produce huge, unpredictable output differences (the avalanche property, exploited by differential cryptanalysis), whereas neural nets use residual connections and LayerNorm specifically to keep gradients smooth and differentiable rather than chaotic.

Reiner Pope contrasts neural nets and ciphers as convergent-but-opposite designs: both scramble information, but ciphers are optimized to blow up small input differences (differential cryptanalysis targets this), while neural nets use residual connections and LayerNorm to keep the system smoothly differentiable for gradient descent.

Reiner Pope · Dwarkesh Podcast

Claim◆Audio · 6:04 · 3m

Fewer than five companies outside of China have all the necessary ingredients to succeed at neural-network-based autonomous driving: a large vehicle fleet for data, control of the perception stack, and the capital for GPUs and custom inference chips.

Scaringe argues that vertically integrated autonomy is the only viable path, and that there are more than one but fewer than five companies (including Rivian, Tesla, and Waymo) that possess the fleet size, sensor control, data architecture, and capital to compete in neural-network-based autonomy.

RJ Scaringe · No Priors

Mechanism◆Video · 34:43 · 2m

The relational foundation model produces predictions via a single frozen forward pass of a transformer-like neural network using in-context learning, with no training, gradients, agent loop, memory, or scratchpad involved.

Jure clarifies that Kumo's relational foundation model is not an agent: it makes predictions through a single forward pass of a pre-trained, frozen transformer-like network with in-context examples, with no backpropagation, memory, or iterative reasoning steps.

Jure Leskovec · The TWIML AI Podcast

MechanismVideo · 17:08 · 2m

Naively chunking documents by a fixed number of characters loses important context, whereas semantically chunking legal documents along their natural structural boundaries (sections, subsections, bullets) while retaining hierarchy preserves accuracy that a naive approach would leave on the table.

Alex explains that Sphere invested heavily in semantic, structure-aware chunking of legal documents rather than naive character-count chunking, because the naive approach sacrifices retrieval accuracy.

Alex Boucott · The TWIML AI Podcast

MechanismArticle · 24 words

The visual bug was fixed by prompting Codex with a direct question about the black spheres followed by a directive to fix it.

The author fixed the eyeball bug with two short follow-up prompts - asking why the raccoons had huge black spheres, then telling it to fix the issue - which resolved the problem.

Author · Simon Willison's Weblog

Claim◆Audio · 60:33 · 2m

A ten-layer neural network can amortize and approximate an almost intractable search problem to very high fidelity — a breakthrough most people still don't fully appreciate, and one that hints our theoretical notions of computational hardness may be incomplete in practice.

Eric Jang argues AlphaGo's core breakthrough — compressing an intractable Go search into a 10-layer forward pass — is more profound than commonly recognized, and that the same phenomenon underlies AlphaFold and may complicate our understanding of NP-hardness.

Eric Jang · Dwarkesh Podcast