Standard RL training rewards an entire trajectory as pass/fail without indicating which specific decisions were good or bad, which is like a teacher grading an essay 'B' without saying what made it a B — attributing credit to the specific pivotal decisions within a trajectory would make RL training far more efficient.
Pullen argues today's RL rewards treat every token in a long rollout as equally responsible for the final outcome, which wastes compute; isolating and grading the specific high-stakes decision points in a trajectory — like a teacher circling the exact sentence that earned a grade instead of just handing back a B — would make training dramatically more efficient. ✦ AI generated
Alistair Pullen · Machine Learning Street Talk · 2026-07-13 · original ↗
starts at this moment · 20:36
“how do you think about slop?”
say you were doing your English A level and you'd written a 2500 word essay on a question and the teacher just gives you right okay that's a B thank you so much and you're like I don't what made it a B... it would be far easier if the teacher just sent, you know, circle the sentence and be like, this is rubbish. Don't say this, right? And that is fundamentally the principle we're trying to bring into like RL across the board.
verbatim transcript · starts at 20:36
20:16certain extent. they they they learn statistical invariences that represent some kind of abstract structure and and that helps them to generalize. And obviously what we want them to do is to learn problems in the abstract so that they can generalize to new novel problems that they've never seen before. So you know the idea is we capture the thought process and then we capture that generalization if we do it well enough.
20:36That that's the rough idea is I mean how do you think about slop? >> So slop is also one of my pet hates. um when you are obviously any any of your viewers who I'm sure use claude code or open code or whatever agentic coding harness they like um see adnauseium is is is like both a combination of model vibe and also slot problems um and you
20:59hit the nail on the head I think in your question which is when you said that okay yeah the test pass right okay okay is it is it technically functionally correct sure right but to at what cost is is the big is the big question and I think fundamentally I mean like when when you think about how these models are trained to do software engineering uh and this is the realization that we
21:19sort of had when we when we built outpost and I'm sure you've you've seen the blogs about how we did it um within that realm normally um and again we don't have that much insight over how this works in in the big labs but normally when you're training a model to be better at software engineering you have some kind of software engineering problem um you have a problem statement
21:41you have some kind of test often a unit test but not always depending on the task type. Um that needs that that is either failing in the early in the prior state and passing in the after state. Um and then you know you you you give the agent the problem. It goes and does its trajectory, its roll out and then at the end you you run the unit test and if it
22:00passes then okay I'm like you got it right. Great. You get a reward and then the the weights update. Um, obviously the the problem with that is is is a few fold, but fundamentally like it could have come up with the most insane way of doing something. Um, whether it be in terms of like commands that it ran that could have been unsafe in terms of code
22:19that's absolute crap compared to what it should have actually done. Um, and all of those things get reinforced whether you like it or not when you give that reward based on purely correctness. So there are a number of things that we have done and are continuing to do um in the RL process to try to ameliate this. Um and in terms of slop specifically there are a couple of key things. One is
- ·Standard RL grades entire rollout pass/fail
- ·Every token treated as equally responsible
- ·No signal on which decisions mattered
- ·This wastes compute during training
- ·Teacher hands back a 2500-word essay: 'B'
- ·Student has no idea what made it a B
- ·Better: circle the exact bad sentence
- ·Isolate pivotal decision points in a trajectory
- ·Grade those specific choices, not the whole run
- ·Goal: bring this credit assignment to RL broadly