Active parameter count, not just total parameter count, is a primary driver of real-world model quality, as shown by a small dense model outperforming a sparse MoE model with far fewer active parameters.
Comparing GPT-OSS 12B (about 5B active params) against the fully dense Devstral 2 123B, Alistair says the dense model feels dramatically better, attributing much of the gap to active parameter count per token. ✦ AI generated
Alistair Pullen · Machine Learning Street Talk · 2026-07-13 · original ↗
starts at this moment · 15:36
“How can we make models that are as good as those frontier models?”
if you take at small scale something like a GPOSS 12B right that is... 5 billion active... and then you take a like devstrol 2 123b which is I believe like a similar architecture to llama 70b so it's fully dense... devstrol feels so much better than the GPOSS model... I think that a huge part of that just comes from the fact that one you have 120 billion active parameters per token, the other you have five.
verbatim transcript · starts at 15:36
15:19dense models, we would >> it would just be strictly better. >> I think it would just be better. Yeah, I think but I think like again that that is taking it to the logical extreme of of >> could you what would you need to inference that? What would you need to actually run a parameter you know a model of that of that size? I think what like an interesting an interesting
15:36example of this is um and and it's not really a fair fight but if you take at small scale something like a GPOSS 12B right that is ane I believe it's 5 billion active I can't remember it was a while ago but it was something like that and then you take a like devstrol 2 123b which is I believe like a similar architecture to llama 70b so it's fully
15:58dense I don't know if you've used them back to back before devstrol feels so much better than the GPOSS model, right? Um, and that that they are architecturally quite quite different. Um, particularly in like the attention mechanism. Um, but fundamentally I think that a huge part of that just comes from the fact that one you have 120 billion active parameters per token, the other you have five and and it is a very we've
16:20deployed both to customers like last year. Um, and the difference was night and day in terms of how it felt and and and what they got out of the coding agent when they were running it. >> Oh, interesting. Yeah, because on the other stuff, so you know, you were talking about like the data, the pre-training, the algorithmic stuff. I mean, maybe the algorithmic stuff is kind of converged only because we now
16:36have this basin of attraction where there's kernel optimizers and entire ecosystems around this and like maybe it's kind of converged, but the data thing is interesting, right? Because like there must be an insane amount of engineering and you know with LLMs, it's a little bit like what's the magic word? Like if you if you frame the question in the right way, it it has that representational friction and and it
16:55does interesting things. So I'm I'm guessing Anthropic do a whole bunch of like you know data curation and and and pruning. But another thing that Anthropic do is they have clawed code. They have the ecosystem and >> they have trajectories coming in all day every day. Yeah. >> Exactly. And those trajectories they because you know you've got this big thing which is that it's not about where
17:12you end up. It's about how you got there and and and I think software engineering is is the it's not about writing code. It's actually like just um creating process. >> Yeah. Creating mental abstractions, doing experiments, refining those abstractions, sharing them with the team. So, so, so this this this process like so iteratively I'm running code, I'm testing things, I'm refining my abstractions and like anthropic have
17:33access to so much of that data. How much of an advantage is that? Obviously, it's a huge advantage. Um, I don't know what their terms of service say. I'm not saying that they train on all that data. I don't know whether they do or not. Certainly one of the interesting things that having trajectories means um and something that like obviously internally we you know collect our own