RL for coding agents wastes efficiency by weighting every token in a trajectory equally based only on a final pass/fail signal; credit should instead be attributed to the specific pivotal decision points, the way a teacher should mark exactly which sentences of an essay were good or bad rather than just returning a single grade.

24:40others in the space I guess are trying to do right now is okay if you can find out what those ranges of like I guess high high entropy tokens or places where you know a decision was made finding that is half the problem and then figuring out once you know that this is an important thing that happened determining whether it was good or bad relative to the final outcome is a

25:00different story. Yeah. Um, but if you can do that a your RL gets significantly more efficient because you're not relying I mean the analogy I always come up with is like okay say you were doing your English A level um and you'd written um you you were practicing and you'd written like an essay for your teacher and you'd written a 2500word essay on a question and the teacher just

25:22gives you right okay that's a B thank you so much and you're like I don't what made it a B like I'm not going to tell you what made it a B it was a B um and Then what you're going to have to do is you're going to have to write hundreds of essays and you'll get an A on some, you'll get a B on others, you get a C on

25:37others, and eventually you are going to be like, okay, so when I do this, I tend to get an A more, so I think this is probably a good thing to reinforce. >> But it would be far easier if the teacher just sent, you know, circle the sentence and be like, this is rubbish. Don't say this, right? And that is fundamentally the principle we're trying to bring into like RL across the board

25:56because you get so much more performance. you get more out of the flops that you have. Um, and also you're teaching the model to learn the things that are actually important and not just the filler, right? >> I know. I mean, the the great thing about machine learning is that it just generalizes low down the abstraction mountain. So, from very superficial statistical generalizations, the bad thing about machine learning is it

26:16generalizes. So, so yeah, I completely agree with you. You know, we we have a huge problem with benchmarks and machine learning. So you know we we are obsessed with pass one pass at five accuracy and we don't seem to care about reliability about you know consistency about security about abstraction forming that's clearly the most important thing I mean France he he he did the arc challenge and unfortunately they were

26:36brute forceable and now now he's got this new version which is so difficult to brute force you have to form abstractions to get any kind of good performance on it but so so you're saying there's a new form of RL perhaps different from the deepseat type type of RL which is like rather than just being rewarded for getting the right answer you're forcing it to form reusable

26:53abstractions and go higher up the mountain. >> Yes, in short, you've explained that far better than I did, but yes, that is that is that is essentially what we're trying to get to. One of the nice things is that I think that essentially what we're talking about here is credit attribution within a within a trajectory. Um, that ports quite nicely to a bunch of different RL algorithms that are in