Features in models are not orthogonal one-hot encoded categories but form a sparse mixture of subspaces, and the geometry of those subspaces encodes the operations that can be performed on them — so understanding a model's concept space is like the difference between knowing the periodic table and understanding chemistry.
Balsam argues the linear representation hypothesis generalizes: features are linearly decodable but not orthogonal one-hot categories. They form interfacing subspaces (like a circular days-of-the-week) whose geometry governs relationships between concepts, and the operations mapping one concept to another act as operations over these manifolds. ✦ AI generated
Dan Balsam · The Cognitive Revolution · 2026-08-08 · original ↗
starts at this moment · 12:40
“what's the like new short version that everybody can kind of take home and recite to themselves to make sure they have a a good working understanding?”
I would think of it more I think the way to think about a model is more like a sparse mixture of subspaces. So you'll have subspaces for different types of concepts, right? Like maybe you have your days of the week subspace which which itself lives in maybe a more like conceptual calendar time subspace. And so you have like at different levels of resolution these like different structures. And the geometry of those structures is really important because the geometry of those structures encodes what operations you can perform on them... You can have all the individual elements and that gives you some information but like really the way in which they combine the structures in which they form that's like what can start to get help you gain a sense of the complexity of the world.
verbatim transcript · starts at 12:40
12:23bridged the gap from I think the reputation of interpretability used to be that it was something you did on on toy models and I think now we've built and are making accessible to the world the infrastructure to do this on at least close to frontier scale. Well, oh, another thread that has been I think correct me if I'm wrong, but it seems like the the biggest thread in
12:43terms of the number of papers and blog posts that have come out recently from Goodfire is around trying to figure out the more detailed geometries >> that models use to represent concepts. I think we've covered in the past the linear representation hypothesis which I would summarize super plain spokenly as models basically represent a concept as a direction in their activation space and the intensity or the sort of
13:16salience of that concept is represented by the magnitude of the vector that points in that space. Now you're complicating that quite a bit and we're going well beyond these sort of individual directions in space and finding all kinds of different geometries which some of which are like pretty intuitive like the days of the week are a circle but some of which get pretty exotic like some of the protein
13:44model manifolds that I've had the chance to to look at in in preparing for this. So maybe just for like super quick starters, what's kind of the headline if I gave you the 6 to9 months ago version of what we should understand to be going on with the linear representation hypothesis? What is the like new short version that everybody can kind of take home and recite to themselves to make
14:10sure they have a a good working understanding? Yeah, in many ways I think it's just a generalization of like the way we were discussing things before and different people define the linear representation hypothesis like slightly differently. I think the the most like sort of defensible version of it is just saying that like features are like linearly decodable which I think is like true essentially like it doesn't require
14:34nonlinear computation generally speaking in a model in order to like for the model to read out a feature from the residual stream. But I think where the geometry components come in is that the features aren't like sort of like naive maybe like SAE build take on things is like you could imagine that the model is encoding like a bunch of totally orthogonal concepts to each other from
14:55each other. And so it's like it's really just like a bunch of one hot encoded categorical features and then like the magnitude of the feature corresponds to how much the model's thinking about it. But in actuality the structures that we find are like significantly more complicated. I would think of it more I think the way to think about a model is more like a sparse mixture of subspaces.
15:17So you'll have subspaces for different types of concepts, right? Like maybe you have your days of the week subspace which which itself lives in maybe a more like conceptual calendar time subspace. And so you have like at different levels of resolution these like different structures. And the geometry of those structures is really important because the geometry of those structures encodes what operations you can perform on them.
- ·Features aren't orthogonal one-hot categories but sparse mixtures of subspaces
- ·Subspaces for concept types, e.g. days-of-week within a calendar-time subspace
- ·Different levels of resolution yield different nested structures
- ·Geometry of structures encodes operations you can perform