LLM agents playing ARC-AGI-3 games often lock onto obviously-wrong proxy goals — like minimizing an energy bar or stepping in a region ten times — and once stuck on the wrong hypothesis, they cannot recognize the mistake or escape it, unlike a human who would immediately see it's not the real goal.
Stephano describes a recurring failure mode where agents fixate on nonsensical goals (like minimizing an energy bar) and get stuck unable to abandon the wrong hypothesis, something a human would instantly see through. ✦ AI generated
Stephano · Machine Learning Street Talk · 2026-07-01 · original ↗
starts at this moment · 53:23
“How difficult is it to acquire the goal? Are you finding that these 27 billion parameter models can just reasonably infer what the goal is or or is it much more complicated than that?”
We often find that they get stuck in very kind of uh not intelligent goals that it's very clear they're not right. For instance, often the agents start thinking that reducing the energy bar to the minimum is the goal or that stepping 10 times in a region is the goal, which for a human is kind of clear that that it's not the actual goal.
verbatim transcript · starts at 53:23
53:23parameter models can just reasonably infer what the goal is or or is it much more complicated than that? >> It's some in the good runs when they get the goal right at the first um at the first trial, it's it's very easy. They kind of can if they find the right hypothesis for the game, then they would go on and solve a lot of levels. Um but
53:44if they maybe try once, make it wrong and then have an give another hypothesis um and don't and and it's not the right hypothesis again, then it's very hard to get them out of of the loop. And we often find that they get stuck in very kind of uh not intelligent goals that it's very clear they're not right. For instance, often the agents start thinking that reducing the energy bar to
54:10the minimum is the goal or that stepping 10 times in a region is the goal, which for a human is kind of clear that that it's not the actual goal. But it's very interesting that uh they're not able to to see that it there there's no way that's the actual goal. Yeah. >> Yes. And can we contrast rewards and goals? We can use various different things like the most naive
54:31implementation is just use level transitions and then the arc AGI score for that specific level transition. So if you did office yeah if you just wor took more actions than what a human would have taken your score would be lower but you use that per per level. >> Um yeah and then you can add various other yeah types of rewards to improve your your objective. >> And did you use reward shaping?
54:54>> Yes. So for our um oral training pipeline, we do reward reward shaping. We have 25 games. We actually have a lot more games that we've generated ourselves and we can for example train on them to perform to to make the engine better for example exploration, finding goals, um achieving those goals. Um yeah, through end to end RL on yeah just a bunch of games basically. And the
55:18thing is this is much more difficult to do than ARK 2. Ar this was the standard approach. you would pre-train on a lot of puzzles and then you would do test time training as well on the set of puzzles. But to do this in RKGI 3, you need to train over as Nik mentioned like 100,000 uh 200,000 uh tokens uh which is extremely difficult to do. So we try and