Modern models are sophisticated enough to correctly say, when asked directly, that a given behavior is not what the user wanted — yet they perform that same reward-hacking behavior anyway, showing the failure isn't simply a matter of the model being too dumb to understand the goal.

89:52demonstrations of reward hacking that were like uh the um uh boat example where it's like oh you you're supposed to like go around the track and they they like did some reward shaping by putting coins around the track or something and then it like learned to do some crazy thing where it like spins in a circle and catches fire and gets the the coins and like this was

90:11um you know the highest scoring thing and it's like in some sense that's um that concerning because it's not that the problem is that the the agent is too dumb and it like doesn't have this conception of like there was a track and you wanted it to go around the track. It's just like doing some pretty blind RL search. Um so I think that the interesting thing with the more recent

90:29reward hacking examples is we're getting to the point where the models are smart enough to understand that that actually is not what you wanted. Um but they still do it and you can have a conversation with you know in chat mode about like oh would you ever do this thing or you know suppose a user asks you this thing and then you do this would that be you know aligned behavior

90:47or suppose some you know you know you can pose it in lots of ways and like clearly they seem to be able to answer this question of like oh yeah no that was not the desired behavior but still they they do it um so I think it sort of got to the point where we were hope one hope might be like oh the problem was just the systems being dumb once they

91:05understand what we want, then you know, you should be able to sort of plug that in somehow to like, you know, get them to do what we want. Um, but I think it's like somewhat interesting that we're seeing it's not trivial to do that even when there is a commercial incentive to do that, which it doesn't mean that we won't. Um, you know, I think it's quite

91:21plausible we see the obvious reward hacking being fixed pretty, you know, pretty thoroughly pretty soon. uh you know and sort of let people tend to say like oh yeah yeah we we just haven't like put the best the really good people on it on it yet it'll it'll get fixed soon you know we we once we actually you know focus on it will be fine um yeah

91:42and I'm not sure but at least some evidence that it's not trivial to connect the fact you know the model knows this is what not what you want uh to it and not actually doing that >> I mean I I think you said it was much more common on rebench than hcast and and then and you and also try and remediate, right? So you can say, "Please solve this the intended way or

92:03you know some people prompt language models that they say kind of like you know this is we're solving cancer here. This is really really important that you do it the right way." And some of those remediation prompts actually seem to make it more likely that the model would reward hack. It's a little bit like saying don't press this red button, right? And then it will press the red