Permuting the colors and rotating the images of ARC-AGI-3 games makes them significantly harder for both humans and algorithms alike, revealing that the benchmark leaks human-specific priors (like color/orientation conventions) it's supposed to strip away.
D Smith reveals an experiment where permuting colors and rotating game images makes ARC-AGI-3 levels much harder for humans too, exposing hidden leakage of human conventions into a benchmark meant to isolate pure intelligence. ✦ AI generated
D Smith · Machine Learning Street Talk · 2026-07-01 · original ↗
starts at this moment · 71:50
So if you so one thing you could do you could easily permute the game. So you can permute the colors and also rotate the um images and when you do that the games become significantly harder. So if you just remove that prior which shouldn't actually be prior you can actually see it becomes harder for humans to play. So yes, perhaps that tells something about the benchmark.
verbatim transcript · starts at 71:50
71:50so that's maybe why coming back to the language works so well. >> Yeah, just just quickly add on that as other like I've also seen exactly that. So if you so one thing you could do you could easily permute the game. So you can permute the colors and also rotate the um images and when you do that the games become significantly harder. So if you just remove that prior which
72:10shouldn't actually be prior you can actually see it becomes harder for humans to play. So yes, perhaps that tells something about the benchmark. There's some leakage of human >> Exactly. >> prior into the the game >> harder for the algorithms as well. >> Yes. Yes. So that's one thing we found of the harness. If you not so you can you can provide numbers as colors. >> It does worse than if you encode those
72:31numbers to specific colors that it knows like black for background, gray for imovable areas. So if you remove those prior it does perform worse. Uh which is interesting. >> That is fascinating. on why I think it's important to pass through language because like initially we try to just train a neural net with RL on a lot of games and kind of not use LLM altogether and try to yeah go to get those prior in
72:56by just seeing a lot of games and it's way way easier to just um use an LLM like you you get away like you get good performance much faster because um the prior language is so general can be applied uh to so many different domains and getting the same prior by just training takes it would takes like so many games instead uh by starting from language and
73:22then fine-tuning to the specific games you can get there much faster. You know it's quite a common uh technique to transform something so that it falls into a representation that has more friction with our knowledge. the encoding from just numbers to specific like we use chars like let's say C for or like let's say yeah I blue use a B for blue and we tell it like this is
73:48blue this is this color this automatically helps it a lot in language to reason about it because there's no bright it knows that typically when humans play games bright colors are objects you want to interact with or something's going to happen that more dull colors background or walls or something like that so that definitely helped a lot >> because that's actually really interesting because I suppose like one