Active parameter count per token, not raw total parameter count, is the main driver of how good a model feels in practice — a fully dense ~123B model with far more active parameters clearly outperforms a sparse MoE model with a similar or larger total size but only ~5B active.
Comparing GPT-OSS-120B (roughly 5B active parameters) against the fully-dense Devstral-123B, Alistair says the difference in real-world feel is night and day, illustrating that active parameters per token matter more than total model size. ✦ AI generated
Alistair Pullen · Machine Learning Street Talk · 2026-07-13 · original ↗
starts at this moment · 15:58
“What would you need to actually run a parameter you know a model of that of that size?”
if you take at small scale something like a GPOSS 12B right that is ane I believe it's 5 billion active I can't remember it was a while ago but it was something like that and then you take a like devstrol 2 123b which is I believe like a similar architecture to llama 70b so it's fully dense I don't know if you've used them back to back before devstrol feels so much better than the GPOSS model, right?
verbatim transcript · starts at 15:58
15:58dense I don't know if you've used them back to back before devstrol feels so much better than the GPOSS model, right? Um, and that that they are architecturally quite quite different. Um, particularly in like the attention mechanism. Um, but fundamentally I think that a huge part of that just comes from the fact that one you have 120 billion active parameters per token, the other you have five and and it is a very we've
16:20deployed both to customers like last year. Um, and the difference was night and day in terms of how it felt and and and what they got out of the coding agent when they were running it. >> Oh, interesting. Yeah, because on the other stuff, so you know, you were talking about like the data, the pre-training, the algorithmic stuff. I mean, maybe the algorithmic stuff is kind of converged only because we now
16:36have this basin of attraction where there's kernel optimizers and entire ecosystems around this and like maybe it's kind of converged, but the data thing is interesting, right? Because like there must be an insane amount of engineering and you know with LLMs, it's a little bit like what's the magic word? Like if you if you frame the question in the right way, it it has that representational friction and and it
16:55does interesting things. So I'm I'm guessing Anthropic do a whole bunch of like you know data curation and and and pruning. But another thing that Anthropic do is they have clawed code. They have the ecosystem and >> they have trajectories coming in all day every day. Yeah. >> Exactly. And those trajectories they because you know you've got this big thing which is that it's not about where
17:12you end up. It's about how you got there and and and I think software engineering is is the it's not about writing code. It's actually like just um creating process. >> Yeah. Creating mental abstractions, doing experiments, refining those abstractions, sharing them with the team. So, so, so this this this process like so iteratively I'm running code, I'm testing things, I'm refining my abstractions and like anthropic have
17:33access to so much of that data. How much of an advantage is that? Obviously, it's a huge advantage. Um, I don't know what their terms of service say. I'm not saying that they train on all that data. I don't know whether they do or not. Certainly one of the interesting things that having trajectories means um and something that like obviously internally we you know collect our own
17:58trajectories of our own use of cosine obviously we don't have anyone else's because of the way we deploy it but for cosine employees using cosine at least um we do still get a fair number of trajectories nothing like the order of magnitude that anthropic get but it the most useful thing for trajectories for us is seeing how users prompt models. So like there's one thing when you're