Video · 2026-07-01 · 1h 25m · 6 moments

ARC-AGI-3 winning team - Millennia of minds, compressed into words.

✦ AI generated

GUEST D Smith GUEST Host GUEST Jon Kotar GUEST Stephano GUEST Michael HOST Tim Scarfe

timeline · colored by role

Definition

The concept of a 'maze' isn't a native primitive present in a newborn's mind — it's an abstraction humans have built up over generations of problem-solving, and injecting such accumulated abstractions into an LLM lets it skip many levels of abstraction that would otherwise have to be learned from scratch.

Michael distinguishes innate core-knowledge priors from generationally-accumulated concepts like 'maze,' arguing that giving LLMs access to these accumulated human abstractions lets them bypass levels of abstraction-building.

transcript

Michael: In the example that we saw we have a maze, right? There's no native primitive in a in a newborn that says this is a maze. But this has been over generations of problem solving through humans uh very much established concept. And if we can inject this into an LLM, we can skip many levels of abstraction.

Anecdote

A professional esports friend recognized the goal of an ARC-AGI-3 game instantly and finished the first level in three seconds without a single wasted move, showing that rapid human mastery of these games comes from bootstrapping on years of prior gaming pattern-recognition rather than solving from scratch.

Jon Kotar recounts showing an ARC-AGI-3 game to a professional esports player, who solved the first level with superhuman efficiency almost instantly — illustrating how much human 'general' problem-solving actually leans on transferable prior experience.

transcript

Jon Kotar: I have a friend that uh is a professional esports player. I showed him one of the games and he completed the first level without spending an extra move that he didn't have to. It was immediate to him what the goal was. He recognized the pattern and I showed him the game. Within 3 seconds, he finished the first level with sub uh super human performance, let's say.

Definition◆

ARC-AGI-3 introduces genuine agency into the ARC benchmark series — defined as the ability of an agent to have goals, plan, and realize them — going beyond the low-level 'sense and act' notion of an agent, toward a cognitive-science notion of future-directed control.

The host argues that ARC-AGI-3, unlike the more abstract ARC-AGI-1/2, centers on 'agency' — having ambitious goals and the planning capacity to realize them — distinguishing it from a merely reactive sense-act loop.

transcript

Host: So agency in my in my definition is the ability of an agent to have goals to plan and realize those goals. And the more ambitious the goals are and the more you realize the goals the more agency you have. So there's a kind of low-level nononsense definition of agency which is just a thing that can sense and act.

Data◆

The oft-cited 36% ARC-AGI-3 score is misleading because it doesn't measure the fraction of games solved — it measures action efficiency, the ratio of a human baseline's action count to the AI's action count (squared), so an inefficient-but-successful solver still scores near zero.

D Smith explains that the headline 36% score on ARC-AGI-3 doesn't mean 36% of games were solved — it's an action-efficiency ratio versus a human baseline, so models solving far more games than 36% still score low if they're inefficient.

transcript

D Smith: 36% might be misleading as a number if you don't look behind it. So what it really measures is action efficiency. So, uh, correct me if I'm wrong, but it's the ratio of the of the human baseline divided by the number of actions, uh, the AI has taken, whatever model it is on the level or a human, whatever whoever the player is, and then squared as well.

extends · 1gives example · 1

Example

LLM agents playing ARC-AGI-3 games often lock onto obviously-wrong proxy goals — like minimizing an energy bar or stepping in a region ten times — and once stuck on the wrong hypothesis, they cannot recognize the mistake or escape it, unlike a human who would immediately see it's not the real goal.

Stephano describes a recurring failure mode where agents fixate on nonsensical goals (like minimizing an energy bar) and get stuck unable to abandon the wrong hypothesis, something a human would instantly see through.

transcript

Stephano: We often find that they get stuck in very kind of uh not intelligent goals that it's very clear they're not right. For instance, often the agents start thinking that reducing the energy bar to the minimum is the goal or that stepping 10 times in a region is the goal, which for a human is kind of clear that that it's not the actual goal.

gives example · 1

Data

Permuting the colors and rotating the images of ARC-AGI-3 games makes them significantly harder for both humans and algorithms alike, revealing that the benchmark leaks human-specific priors (like color/orientation conventions) it's supposed to strip away.

D Smith reveals an experiment where permuting colors and rotating game images makes ARC-AGI-3 levels much harder for humans too, exposing hidden leakage of human conventions into a benchmark meant to isolate pure intelligence.

transcript

D Smith: So if you so one thing you could do you could easily permute the game. So you can permute the colors and also rotate the um images and when you do that the games become significantly harder. So if you just remove that prior which shouldn't actually be prior you can actually see it becomes harder for humans to play. So yes, perhaps that tells something about the benchmark.

Highlight slides

ARC-AGI-3's Core Claim✦ from: ARC-AGI-3 introduces genuine agency into the ARC benchmark series — defined as the ability of an agent to have goals, plan, and realize them — going beyond the low-level 'sense and act' notion of an agent, toward a cognitive-science notion of future-directed control.Two Notions of 'Agent'✦ from: ARC-AGI-3 introduces genuine agency into the ARC benchmark series — defined as the ability of an agent to have goals, plan, and realize them — going beyond the low-level 'sense and act' notion of an agent, toward a cognitive-science notion of future-directed control.Why It Matters✦ from: ARC-AGI-3 introduces genuine agency into the ARC benchmark series — defined as the ability of an agent to have goals, plan, and realize them — going beyond the low-level 'sense and act' notion of an agent, toward a cognitive-science notion of future-directed control.36% Doesn't Mean Solved✦ from: The oft-cited 36% ARC-AGI-3 score is misleading because it doesn't measure the fraction of games solved — it measures action efficiency, the ratio of a human baseline's action count to the AI's action count (squared), so an inefficient-but-successful solver still scores near zero.How the Score Is Calculated✦ from: The oft-cited 36% ARC-AGI-3 score is misleading because it doesn't measure the fraction of games solved — it measures action efficiency, the ratio of a human baseline's action count to the AI's action count (squared), so an inefficient-but-successful solver still scores near zero.Why It's Misleading✦ from: The oft-cited 36% ARC-AGI-3 score is misleading because it doesn't measure the fraction of games solved — it measures action efficiency, the ratio of a human baseline's action count to the AI's action count (squared), so an inefficient-but-successful solver still scores near zero.

Across the web

17 comments — Viewers debate an AI/ARC-AGI research video's ideas on intelligence, learning, and benchmarks while praising the hosts, guests, and production quality. ✦ AI summary

Being a kaggler i have seen the work of Jeroen Cottaar, the guy is just amazing. Seriously wanted to work under him or get some mentorship from him.

youtube · @arpitbansal4911 · ▲ 10 · 2026-07-02 · source ↗

What camera are you using for this? Looks amazing

youtube · @0113TrenSetter · ▲ 5 · 2026-07-02 · source ↗

Brilliant video! I love where you talked about using priors to bootstrap. Like the part about the new hire and the gamer friend. Organisms have to learn new things in their own way. They have to map new situations to priors. "AGI" will need to train on the full fidelity distribution of reality, using its own successes and mistakes to create its own priors and build causal graphs.

youtube · @dsw7441 · ▲ 3 · 2026-07-03 · source ↗

This can amaze me; ask a human to “come up with 100 random names, assign an age between 20 and 100, and then sort all of the names in alphabetical order, get an average age for those over 30, and then count the R:s appearing”.. it’s like; if we ask a human to do that without a tool, even paper and pen; and then we are surprised for a single error. Anyhow; when AI has tool calling it too have a sort of paper & pen; and when nudged to just evaluate the results with some self-made tests; it’s basically 100% accurate every time. When it comes to guessing time; same thing; just allowing a ping back by a “cron” at a later time it’s always accurate. If we ask a human to do all of this without any tools or errors we normally expect some others quirks, such unable to tie one’s shoelaces, don’t we. Hmm .

youtube · @kilianlindberg · ▲ 3 · 2026-07-03 · source ↗

"It has to discover rules". -- It's quite an interesting way to put it! Wasn't ML as a paradigm all about letting algorithms discover patterns (rules) at its core, from the start?

youtube · @Anton_Sh. · ▲ 3 · 2026-07-02 · source ↗

For me the ultimate test is if you put the system inside a physical robot body, and the system is able to move correctly, understand it's surroundings and complete the tasks that are completable with that body. Also we are focusing much on visuals, but there are also sound, touch and propioception dimentions that aren't being explored. What would a purely sound driven game look like?

youtube · @vladyskaizen · ▲ 2 · 2026-07-02 · source ↗

When Fable was launched for the first time it wasnt there for even a week which is needed to run these tests, With fable dropping again tommorow, it will be exciting to see how it performs on ARC AGI

youtube · @Aniket_13wb · ▲ 2 · 2026-07-02 · source ↗

These are two of my most favorite teams, one is Tufa and the other is Sakana. Both are filled with brilliant people. Not sure what's the minimum to apply to tufa?

youtube · @PankajDoharey · ▲ 1 · 2026-07-03 · source ↗

1:16:40 with an assumption that intelligence == compression, what we call "language" today is just an ultimate compression layer regardless of signal source. Complex enough image to action model that passes ARC-3 would likely develop compressed representation similar to language in levels of compression and abstraction in order to solve problems.

youtube · @vslaykovsky · ▲ 1 · 2026-07-03 · source ↗

MLST has such high quality videography and great guests. The host however likes to imply that his hunches are fact. That’s a bit unscientific for my taste

youtube · @xt-89907 · ▲ 1 · 2026-07-02 · source ↗

Ohh yes, there's an elephant in the room, and it's most certainly an elephant. We really should stop arguing about its elephant-ness or non-elephant-ness because it's an elephant.

youtube · @wwondertwin · ▲ 1 · 2026-07-03 · source ↗

Damn, the video is so damn good I missed much watching again.

youtube · @PankajDoharey · ▲ 0 · 2026-07-03 · source ↗

Why do you work on harness and not on world models with hierarchical planning ? Does the exploration phase involve the encoding of monoamines or neurotransmitters?

youtube · @tom-et-jerry · ▲ 0 · 2026-07-04 · source ↗

It's Dr. Dries Smit btw :)

youtube · @nina_wiese · ▲ 0 · 2026-07-04 · source ↗

Backpropagation does not occur biologically; instead, it is an energy-based model where weights are adjusted as follows: the model receives input data from several different sensory sensors that are observing the same event or events. The various networks resonate based on the correlation. I haven't yet figured out how.

youtube · @tom-et-jerry · ▲ 0 · 2026-07-04 · source ↗

how many dollars to train up a baby to complete ARC AGI 3 $100,000?

youtube · @blengi · ▲ 0 · 2026-07-04 · source ↗

Is someone going to explain what ARC-AGI is for the newbies?

youtube · @doloresabernathy9809 · ▲ 0 · 2026-07-02 · source ↗

Related episodes

Audio · Dwarkesh Podcast

Eric Jang – Building AlphaGo from scratch

→

Article · The Pragmatic Engineer

Tech interviews with NeetCode

→