Standard RL treats every token in a long trajectory as equally responsible for the final pass/fail reward, which is nonsensical because only a handful of pivotal decision points actually determined the outcome — a flaw that helps produce slop.
Alistair argues that current RL for coding agents rewards entire trajectories uniformly based on the end result, which he calls 'insane' since only a few high-stakes decisions in a rollout actually mattered — motivating Cosine's work on credit attribution to reduce slop. ✦ AI generated
Alistair Pullen · Machine Learning Street Talk · 2026-07-13 · original ↗
starts at this moment · 23:49
“So, so, so this this this process like so iteratively I'm running code, I'm testing things, I'm refining my abstractions... so how do you think about slop?”
fundamentally this notion of you have a roll out of maybe 256,000 tokens, right? In some extreme cases. Um and that culminated in like a one or a zero depending on what the model did. And what we're saying at the moment in many cases is okay, all of those tokens are equally weighted in getting us to that answer.
verbatim transcript · starts at 23:49
23:49think Andre Kapathy said this like over a year ago, but fundamentally this notion of you have a roll out of maybe 256,000 tokens, right? In some extreme cases. Um and that culminated in like a one or a zero depending on what the model did. And what we're saying at the moment in many cases is okay, all of those tokens are equally weighted in getting us to that answer. Yeah.
24:12>> Which when you think about it is insane because that's clearly not true. Um and in so many cases um the that there will be small decision well or like important decisions in a trajectory that were sort of you know forks in the road that would have either that could have resulted actually a bad outcome um where the model decided to go down the right one and ended there. Um and what we and
24:40others in the space I guess are trying to do right now is okay if you can find out what those ranges of like I guess high high entropy tokens or places where you know a decision was made finding that is half the problem and then figuring out once you know that this is an important thing that happened determining whether it was good or bad relative to the final outcome is a
25:00different story. Yeah. Um, but if you can do that a your RL gets significantly more efficient because you're not relying I mean the analogy I always come up with is like okay say you were doing your English A level um and you'd written um you you were practicing and you'd written like an essay for your teacher and you'd written a 2500word essay on a question and the teacher just
25:22gives you right okay that's a B thank you so much and you're like I don't what made it a B like I'm not going to tell you what made it a B it was a B um and Then what you're going to have to do is you're going to have to write hundreds of essays and you'll get an A on some, you'll get a B on others, you get a C on
25:37others, and eventually you are going to be like, okay, so when I do this, I tend to get an A more, so I think this is probably a good thing to reinforce. >> But it would be far easier if the teacher just sent, you know, circle the sentence and be like, this is rubbish. Don't say this, right? And that is fundamentally the principle we're trying to bring into like RL across the board
- ·Rollouts can span 256,000 tokens
- ·Outcome collapses to a single 1 or 0
- ·Every token gets equal credit or blame
- ·Only a handful of decisions actually mattered
- ·Uniform weighting ignores pivotal moments
- ·Flawed credit signal helps produce slop
- ·Building credit attribution into RL training
- ·Goal: reward the decisions that counted
- ·Aims to reduce slop in coding agents