The oft-cited 36% ARC-AGI-3 score is misleading because it doesn't measure the fraction of games solved — it measures action efficiency, the ratio of a human baseline's action count to the AI's action count (squared), so an inefficient-but-successful solver still scores near zero.

49:24hard to of course want a lot more compute than the human beating these games is spending. So yeah, does that really count as the old models can play games now if if it takes them so much effort? I would say that is a real gap and also let's say being able to do this frontier models of no sorry with small open source models would demonstrate a lot more which indeed what we're doing

49:43in the capital competition >> 36% might be misleading as a number if you don't look behind it. So what it really measures is action efficiency. So, uh, correct me if I'm wrong, but it's the ratio of the of the human baseline divided by the number of actions, uh, the AI has taken, whatever model it is on the level or a human, whatever whoever the player is, and then

50:08squared as well. So, this plays really adversarily to anything that's a little bit action inefficient. So, 36% actually in this case doesn't mean we solve that approach solves 36% of the games. it solves way more of the games I would from the training set at least this number is on uh but it just solves them inefficiently. So I think this is really important to emphasize that like the

50:32current frontier models are able to solve like what is it something like half of twothirds of the training games actually till the end but just not as efficiently. >> What is the hardest thing in ARC AGI3? Is it the goal acquisition or is it just simply the action efficiency >> from what we see on the training set? So the the testing set uh the private set is set to be harder. We don't know

50:54anything about it. No, nobody outside of the organization has their organization has seen it. Uh so we we don't know how how hard it is. But as at least on the training games, we see that the that the LLMs can acquire the correct goals and pursue them u somewhat effectively. Um the question is whether this holds also for the for the withheld private test but um it seems that goal setting is not

51:24the bottleneck. So rather the action efficiency and the accumulation of the knowledge over a very long context because you need millions hundreds of thousands if not millions of tokens to solve this and keeping consistent knowledge of everything that has happened over such a context is a is a major engineering challenge at this moment. What is um harder in ARV3 compared to the other one is this interplay between exploration and