Video · 2026-08-08 · 1h 57m · 6 moments

Thinking in Silico: Goodfire CTO Dan Balsam on Concept Manifolds & a $1000/Month ML Research Agent

✦ AI generated

timeline · colored by role

Mechanism◆

Models do not acquire most of their knowledge or capabilities during post-training; post-training (including RL) mostly makes low-likelihood events learned in pre-training more likely, so it only produces small nudges to the weights, and effective intervention must target what data is reinforcing.

Balsam explains that a model's knowledge and capabilities come primarily from pre-training, and post-training only nudges the weights by making certain low-likelihood behaviors more likely. This premise underpins predictive data debugging: the features active when a model reads data predict what that data will reinforce.

transcript

Dan Balsam: there's a good amount of evidence that models don't most of what a model knows like most of its sort of like knowledge and capabilities come from pre-training what happens in post-training including like RL is mostly making low likelihood events from pre-training more likely and so there's not that much new knowledge or like sort of new capabilities that get baked into models in post-training process... because there's only like sort of relatively small nudges in the weights that are happening in any type of post-raining process, most of the sort of like raw capability to produce whatever outcome postrain is going to do already exists in the model in some way.

Claim

Data filtering and reward shaping for off-target feature learning are deeply isomorphic: they achieve approximately the same effects and reach 'two sides of the same mountain,' so researchers can choose either filtering the data or intervening on the training process when predictive data debugging reveals an unwanted effect.

Balsam highlights what he calls his favorite result from the predictive data debugging paper: that data filtering and activation-based reward shaping are two sides of the same coin, producing approximately equal effects on off-target learning, giving practitioners two roughly equivalent options.

transcript

Dan Balsam: I think like my favorite thing from that paper was that like the researchers involved in that paper like show that there's like sort of a pretty deep isomorphism between those two things. Like they they kind of are two sides of the same mountain like filtering the data and reward shaping and they achieve like approximately the same effects and approximately the same amount of offtarget effects as each other.

Mechanism◆

Features in models are not orthogonal one-hot encoded categories but form a sparse mixture of subspaces, and the geometry of those subspaces encodes the operations that can be performed on them — so understanding a model's concept space is like the difference between knowing the periodic table and understanding chemistry.

Balsam argues the linear representation hypothesis generalizes: features are linearly decodable but not orthogonal one-hot categories. They form interfacing subspaces (like a circular days-of-the-week) whose geometry governs relationships between concepts, and the operations mapping one concept to another act as operations over these manifolds.

transcript

Dan Balsam: I would think of it more I think the way to think about a model is more like a sparse mixture of subspaces. So you'll have subspaces for different types of concepts, right? Like maybe you have your days of the week subspace which which itself lives in maybe a more like conceptual calendar time subspace. And so you have like at different levels of resolution these like different structures. And the geometry of those structures is really important because the geometry of those structures encodes what operations you can perform on them... You can have all the individual elements and that gives you some information but like really the way in which they combine the structures in which they form that's like what can start to get help you gain a sense of the complexity of the world.

Claim

Steering along the manifold significantly outperforms steering off it: naive contrastive vectors cut through between-concept regions that are out-of-distribution to the model, while following the learned geometric curve (e.g., the days-of-the-week circle) allows smooth interpolation and avoids the degradation-to-gibberish failure of ordinary steering.

Balsam explains the practical payoff of manifold geometry: early 'Ember'-style steering failed because it cut through the middle of concept structures that are off-manifold and meaningless to the model. Steering along the recovered manifold enabled better protein-model control, such as adjusting the number of beta-propeller blades, and prevents steering collapse.

transcript

Dan Balsam: we've shown that steering along the manifold, intuitively this makes sense, is way better than steering off the manifold. If I have the days of the week in a circle and I want to get from Monday to Friday, the naive way if you're just taking a contrastive vector or something like that is you're going to cut through the middle of the circle. But to the model, the middle of the circle doesn't mean anything. The middle of the circle is not a day... But if I can follow the circle, then I can smoothly interpolate between the different days of the week... And when we smoothly extrapolate on these characteristics, we're actually able to change them without fundamentally leading to degradation in the model.

explains mechanism · 1

Definition

Black sparse featurizers are a generalization of sparse autoencoders in which each feature is a small vector/network rather than a single scalar, recovering semantically meaningful subspaces unsupervised — in some cases the recovered activation structure itself tracks the 3D structure of the represented object (like a walking wolf's body and tail).

Balsam describes BSFs as effectively an SAE and a neural network 'having a baby': each location in the long sparse vector is itself a small network, letting features be multidimensional and avoiding SAE pathologies. In image models this can recover rich structures, e.g., a rabbit feature splitting into ears and face, and activation subspaces that literally wiggle along with a walking wolf in 3D.

transcript

Dan Balsam: I think the easiest way to think about it is yeah like it's a generalization of an SAE where an SAPE assumes that features are onedimensional and instead you just don't have to do that. So instead of a scaler for every feature you can have a vector for every feature and some tricks in the machine learning that you can do in order to make sure that this like trains correctly and it and it learns. But what we find on various models is that this is successful in in recovering in an unsupervised way semantically meaningful subspaces and we find that features are much richer than we may have otherwise seen and they don't suffer from some of the same pathologies sapes suffer from.

Claim◆

Every model is effectively a sparse mixture of experts in which only a tiny percentage of weights matter for any given forward pass, so interpretability is fundamentally about factoring a model — and the unresolved hard problem is not factoring but refactoring, i.e., training or intervening so the model is built correctly in the first place rather than patched post hoc by steering.

Balsam argues interpretability is about factorization: only a small subnetwork matters for each prediction, and Goodfire can now attribute behaviors to single neurons (e.g., a mis-calibrated drink-counting neuron in the WeirdChat dataset fixable by steering). But steering is 'cheating' as a causal proof; the real open challenge is refactoring — producing models whose mechanisms are correct from the start, which is how interpretability generalizes to alignment.

transcript

Dan Balsam: I think the the argument that of the parameter decomposition line of work is that really every model is a sparse mixture of experts and you just have to and over any given forward pass a very very small percentage of the weights actually matter... interpretability is all about factoring... And I think it's all like at the end of the day like interpretability is all about factoring... But I think the thing that's still like really missing is the question of like, okay, I can factor a codebase, but how do I refactor the codebase? How do I put things together back together better than I found them? And on some level like steering I think is is cheating as a solution... what I'd really like to do is produce a model where that neuron was firing the right amount in the first place. And so how do you get from your understanding of how one model works to an understanding of how you produce models that do what you want in the first place? And that's sort of how you generalize from interpretability as like a factoring tool to interpretability as a tool for alignment.

extends · 1

Highlight slides

Capabilities Come Almost Entirely from Pre-training✦ from: Models do not acquire most of their knowledge or capabilities during post-training; post-training (including RL) mostly makes low-likelihood events learned in pre-training more likely, so it only produces small nudges to the weights, and effective intervention must target what data is reinforcing.Post-training Only Nudges the Weights✦ from: Models do not acquire most of their knowledge or capabilities during post-training; post-training (including RL) mostly makes low-likelihood events learned in pre-training more likely, so it only produces small nudges to the weights, and effective intervention must target what data is reinforcing.Features as a Sparse Mixture of Subspaces✦ from: Features in models are not orthogonal one-hot encoded categories but form a sparse mixture of subspaces, and the geometry of those subspaces encodes the operations that can be performed on them — so understanding a model's concept space is like the difference between knowing the periodic table and understanding chemistry.Interpretability as factorization of a sparse model✦ from: Every model is effectively a sparse mixture of experts in which only a tiny percentage of weights matter for any given forward pass, so interpretability is fundamentally about factoring a model — and the unresolved hard problem is not factoring but refactoring, i.e., training or intervening so the model is built correctly in the first place rather than patched post hoc by steering.The open problem: refactoring, not steering✦ from: Every model is effectively a sparse mixture of experts in which only a tiny percentage of weights matter for any given forward pass, so interpretability is fundamentally about factoring a model — and the unresolved hard problem is not factoring but refactoring, i.e., training or intervening so the model is built correctly in the first place rather than patched post hoc by steering.

Related episodes

Audio · No Priors

Baseten CEO Tuhin Srivastava on the AI Inference Crunch, Custom Models, and Building the Inference Cloud

→

Article · Interconnects

5 useful things you'll learn in my new post-training textbook (shipping now!)

→

Article · Import AI

Import AI 461: "Alignment is not on track"; FrontierCode; and synthetic research interns

→