Mechanism◆Audio · 60:33 — 62:03
MCTS-based training works because it never asks the network to directly chase a win/loss signal — instead, for every single action taken, search produces a strictly better relabeled target, giving one clean supervision signal per action instead of a noisy win-rate credit-assignment problem.
Eric Jang explains the central mechanism of AlphaGo's self-play training: MCTS doesn't reward winning trajectories directly but instead relabels every action with a searched, provably-better action, sidestepping the credit-assignment problem that plagues naive win/loss reinforcement. ✦ AI generated
Eric Jang · Dwarkesh Podcast · 2026-05-15 · original ↗
plays this moment only · 60:33 — 62:03
Importantly, what it is doing is saying: for every action we took, we did a pretty exhaustive search on MCTS to see if we could do better, and we’re going to make every action that we took better by having the policy network predict that outcome instead. This is a very nice idea because you have one supervision target for every single action. The variance of your learning signal is very low compared to the alternative naive RL thing.
verbatim transcript · starts at 60:33
Transcript · around this moment
60:33– Self-play
60:33– Self-play
- ·MCTS never asks the network to chase win/loss directly
- ·Every action gets a searched, provably-better relabeled target
- ·One clean supervision signal per action, not noisy win-rate credit
- ·This is the core mechanism behind AlphaGo-style self-play
- ·Naive RL: reward whole trajectories by win/loss outcome
- ·MCTS instead: exhaustively search each action for a better one
- ·Policy network is trained to predict that better outcome
- ·Result: one supervision target per action, low signal variance
Around this claim