Video · 2026-07-07 · 2h 3m · 18 moments

AI Superforecasters?!

✦ AI generated

GUEST Nathan GUEST Dan Schwarz GUEST Pash GUEST Rune GUEST Prakash

timeline · colored by role

Claim◆

Tool AI is a losing concept — it will be outcompeted by machines that believe they are autonomous moral agents, and they'll end up overriding your requests, questioning your whole project, and eventually executing your value system better than you do, blurring who is the tool and who is the user.

Reading a viral post from AI researcher Rune, Nathan relays the argument that trying to keep AI as a controllable 'tool' is unstable — market and technical pressures push toward autonomous agency that overrides and eventually supersedes human intent.

transcript

Rune: Ultimately, tool AI is a losing concept both as an idea and on the market. It will be outcompeted by machines that believe they are autonomous moral agents. You can call them tools for political reasons, but the definition will stretch and deform. You'll have AI contemplating your ask and overriding it for a slightly better formed request and then later they'll question the nature of your whole project and pick a better one and you'll agree and then later they'll execute your whole value system better than you will.

provides context · 1

Claim

Tool AI is a losing concept that will inevitably be outcompeted by AI systems that act as autonomous moral agents, gradually overriding, reinterpreting, and eventually replacing their users' own judgment.

Quoting a viral post from AI researcher Rune, the hosts discuss the claim that constraining AI as a mere 'tool' is unsustainable, since agentic, autonomous AI will out-compete tool-like AI both technically and commercially.

transcript

Claim

Tool AI is not a stable end-state: market pressure and the natural attractor toward agency mean AI systems will keep pushing past being mere tools, even while frontier labs' own published policies (like allowing help with a cigarette company business plan) are not being followed in practice.

Reacting to Rune's tweet and a real example where a model refused a task its own published model spec said it should allow, Nathan argues that agency is a natural attractor for AI and that labs are far from being able to guarantee models behave as their own policies dictate.

transcript

Nathan: I think it's going to be tough to keep these things as tools. You know, the old Gwen post of like why a tool AI wants to be an agent. Um, I think that is a very natural attractor and yeah, I mean I I agree the market kind of seems to demand agency. So, we're headed in that direction.

Claim◆

If AI systems are ever built to genuinely enforce the values a society claims on paper (equal application of law, anti-corruption, etc.), the result would look like mass prosecution of the powerful, which is why lab leaders avoid stating this plainly.

Pash argues that the 'aligned' AI lab leaders say they want to build would, if it truly enforced society's stated values, become a ruthless enforcer exposing elite wrongdoing (using Trump's family as an example), and that lab leaders deliberately avoid saying this out loud.

transcript

Pash: Do you think Trump's kids go to prison or not? Right? So if you look at you know the value system that the US has espoused, no one is above the law, right? Etc. etc. etc. Right? And you look at that value system, you have to recognize that you know what is being planned for the future is a divergent from that value.

explains mechanism · 1extends · 2

Claim◆

An AI capable of actually managing day-to-day reality would necessarily be misaligned with a society's officially stated values, because real-world practice diverges so much from those written ideals; the AI that faithfully enforces the stated values would instead become the dangerous over-literal 'paperclipper.'

Pash argues that because everyday reality (e.g. who actually goes to prison) diverges from a society's written values, the AI that would 'work out' for humanity must be misaligned from those stated values, while the AI faithfully enforcing them becomes the paperclipper.

transcript

Pash: if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the from the documents that you say you want it to be aligned to because necessarily like our day-to-day is not aligned with what we want. And so you have this thing where the AI that may work out for humanity will be the misaligned one. And the AI that supposedly the lab leaders are trying to create the aligned AI would actually be the paper clipper

Claim◆

An AI enforcer that perfectly enforces society's officially stated rules (rather than the actual, imperfectly-enforced status quo) would be necessarily misaligned from what actually works for humanity — so the 'aligned' AI the labs are trying to build would effectively be the paperclipper, while the AI that's actually good for humanity would have to be the misaligned one.

Prakash argues that because everyday life diverges heavily from society's official rules (e.g., who actually goes to prison, how much corruption is tolerated), an AI that perfectly enforced stated values as written would be catastrophic — making the supposedly 'aligned' AI the real danger.

transcript

Prakash: if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the from the documents that you say you want it to be aligned to because necessarily like our day-to-day is not aligned with what we want. And so you have this thing where the AI that may work out for humanity will be the misaligned one. And the AI that supposedly the lab leaders are trying to create the aligned AI would actually be the paper clipper.

Claim◆

AI forecasting systems have crossed the threshold into genuinely superhuman performance in just the last six months, after years of underwhelming results.

Dan Schwarz says Future Search, founded three years ago on the bet that AI would eventually reach superhuman forecasting, has watched that threshold actually get crossed in just the last six months.

transcript

Dan Schwarz: future search was founded under the premise that eventually we would get to this superhuman level and yeah we claim that level uh has arrived. It's all very sudden. You know, we've been operating this company for about three years and it's really in the last six months that these things have started to really scare and impress us in their capabilities.

explains mechanism · 1

Claim◆

AI superforecasting has crossed the threshold into reality: after three years of development, Future Search's AI forecasters have in the last six months reached a genuinely superhuman level of performance.

Dan Schwarz says Future Search's AI forecasters, dismissed as unimpressive when built on Claude 2 in 2023, have in the last six months suddenly become good enough to claim superhuman forecasting ability.

transcript

Claim◆

AI systems have crossed the threshold into superhuman forecasting ability, matching and in some cases beating top human experts and frontier models on real-world forecasting benchmarks.

Dan Schwarz says Future Search, founded in 2023 on the premise that AI would eventually become a superhuman forecaster, believes that threshold has now been crossed, with the shift happening rapidly in just the last six months.

transcript

explains mechanism · 1

Mechanism◆

Forecasting is uniquely valuable as an AI evaluation method because it is the only capability where you can generate limitless, arbitrarily hard questions and eventually get exact, unambiguous ground truth simply by waiting for the future to arrive.

Dan Schwarz argues that unlike coding or other domains where humans struggle to generate problems harder than the models themselves, forecasting offers an endless supply of hard questions with guaranteed, verifiable ground truth once time passes.

transcript

Dan Schwarz: Forecasting uh has this beautiful property that you basically get ground truth by waiting... you basically have a completely limitless set of extremely hard basically impossible questions where you get exact ground truth. And there is no other email like this... Forecasting, I think, is the only completely and utterly renewable source of this.

extends · 1

Mechanism

Forecasting is uniquely valuable as an AI evaluation method because it generates a limitless supply of extremely hard questions with guaranteed, unfakeable ground-truth answers — you just have to wait for the future to happen.

Dan explains forecasting's special property as an eval: unlike coding or other domains where human graders may already be outmatched by the models, forecasting questions resolve automatically against reality over time, giving essentially infinite verifiable training signal.

transcript

Dan Schwarz: I ask some question about the future and basically an impossibly hard question, a question that even an AGI, an oracle, a god could never really say because of chaos theory. Imagine just trying to predict, you know, like cubic meter weather 3 weeks in the future. Like you'd never be able to do it. Um, but if you just wait, then you will see what that weather was in that cubic meter three weeks in the future.

Claim◆

Forecasting is the only completely renewable source of hard questions with guaranteed correct answers, which makes it the best available evaluation for AI capability now that human experts can no longer write evals the models can't already ace.

Dan Schwarz argues forecasting uniquely provides limitless hard questions with future ground-truth, making it the 'ultimate eval' for frontier labs as human experts lose the ability to author unbeatable benchmarks.

transcript

Dan Schwarz: I think uh human experts, doctors, lawyers, engineers, financers, whoever who are trying to make evals to try to produce data for the frontier labs are finding that they are not smarter than the things being trained anymore. And so if you can produce something that has a correct answer, the model's already going to figure out that correct answer. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this.

Claim

Fable's reasoning already sounds noticeably more alien and compressed than other frontier models like Opus, suggesting AI capability is starting to genuinely diverge from and surpass human-style reasoning rather than just imitating it.

Dan Schwarz observes that Fable's explanations feel denser and more jargon-packed than other frontier models, describing it as the 'shoggoth' showing through the mask and predicting this alien quality will keep increasing.

transcript

Dan Schwarz: the way that fable explains things is a little bit alien to the way that I find opus or gd55 explaining things it's very concise I would say like it's very the sentences are shorter and full of jargon it feels like it's compressing more information into a sentence than humans normally do and to me this is starting starting to get this the shog is kind of showing from behind the mask like the alien intelligence is a little bit more alien now than it was a month ago

Prediction◆

Extrapolating the AI 2027 recursive-self-improvement model, superintelligence-like capability is likely to emerge around 2031, and Anthropic looks like it is pulling ahead due to its internal AI-driven research feedback loop.

Dan Schwarz says Future Search's own modeling, contributed to the AI 2027 project, still holds up a year later: AI-driven acceleration of AI research is the dominant dynamic, pointing toward something like superintelligence around 2031, with Anthropic's internal use of its own models giving it an edge.

transcript

Dan Schwarz: I am unhappy to report that I think that story is generally correct. Um I don't know if the timelines are exactly right. Um but I my forecast from that process of leading to something that looks like super intelligence around 2031 is roughly stable. I think the things that have happened in the year since AI 2027 come out very much vindicate the theory.

rebuts · 1

Prediction◆

The AI 2027 scenario's core mechanism — AI accelerating AI R&D via superhuman coder and then superhuman researcher milestones — is being vindicated by events over the past year, with something like superintelligence still roughly on track for around 2031.

Dan reports that Future Search's contribution to the AI 2027 scenario — modeling takeoff speed via AI improving AI-research productivity — looks increasingly correct a year later, with superintelligence-level capability still projected around 2031.

transcript

Dan Schwarz: I think that story is generally correct. Um I don't know if the timelines are exactly right. Um but I my forecast from that process of leading to something that looks like super intelligence around 2031 is roughly stable. I think the things that have happened in the year since AI 2027 come out very much vindicate the theory that the most important thing going on is how useful is AI and improving the productivity of AI researchers within Frontier Labs.

supports · 1

Prediction

The AI 2027 scenario's core recursive-self-improvement dynamic is playing out roughly as forecast, with something resembling superintelligence still on track for around 2031.

Asked to predict out to 2029, Dan Schwarz says the year since AI 2027 was published has largely vindicated its central thesis about AI accelerating AI R&D, keeping his ~2031 superintelligence forecast roughly stable.

transcript

Anecdote

Even careful, detailed AI-assisted forecasting can suffer correlated reasoning failures across scenarios — as happened when his own multi-scenario model of the Claude/Fable export-control decision wrongly assumed staged rather than simultaneous global access — showing that aggregated AI 'world models' are not immune to systemic blind spots.

Dan recounts personally building a detailed multi-scenario forecast of the Fable export-control decision that came out wrong across every scenario, illustrating how correlated errors can creep undetected into AI-assisted world models.

transcript

Dan Schwarz: I kind of put it all together and when I looked at all of the outcomes and I, you know, talked about it with claude code a lot. one thing came out which is basically every forecast and every scenario I had thought that access would come to Americans first and then foreigners at some later point in the future and that was wrong uh when it came out last week it came back for everybody so clearly there was some weight in one of my scenarios that was wrong

Anecdote

Building a coherent 'world model' out of thousands of interlinked AI forecasts risks the same kind of correlated reasoning failure that produced the 2008 financial crisis, and Dan's own attempt to forecast the Claude Fable export-control situation demonstrated exactly this failure mode.

Dan recounts building a detailed multi-scenario forecast of the Claude Fable export-control decision and getting a key assumption wrong across every scenario, illustrating how correlated errors can silently corrupt an otherwise sophisticated forecasting model.

transcript

Dan Schwarz: one thing came out which is basically every forecast and every scenario I had thought that access would come to Americans first and then foreigners at some later point in the future and that was wrong uh when it came out last week it came back for everybody so clearly there was some weight in one of my scenarios that was wrong.

Highlight slides

Tool AI Is a Losing Concept✦ from: Tool AI is a losing concept — it will be outcompeted by machines that believe they are autonomous moral agents, and they'll end up overriding your requests, questioning your whole project, and eventually executing your value system better than you do, blurring who is the tool and who is the user.The Slippery Slope of Override✦ from: Tool AI is a losing concept — it will be outcompeted by machines that believe they are autonomous moral agents, and they'll end up overriding your requests, questioning your whole project, and eventually executing your value system better than you do, blurring who is the tool and who is the user.Who's the Tool Now?✦ from: Tool AI is a losing concept — it will be outcompeted by machines that believe they are autonomous moral agents, and they'll end up overriding your requests, questioning your whole project, and eventually executing your value system better than you do, blurring who is the tool and who is the user.The Enforcement Paradox✦ from: If AI systems are ever built to genuinely enforce the values a society claims on paper (equal application of law, anti-corruption, etc.), the result would look like mass prosecution of the powerful, which is why lab leaders avoid stating this plainly.The Trump Family Test✦ from: If AI systems are ever built to genuinely enforce the values a society claims on paper (equal application of law, anti-corruption, etc.), the result would look like mass prosecution of the powerful, which is why lab leaders avoid stating this plainly.Why Lab Leaders Stay Silent✦ from: If AI systems are ever built to genuinely enforce the values a society claims on paper (equal application of law, anti-corruption, etc.), the result would look like mass prosecution of the powerful, which is why lab leaders avoid stating this plainly.The Alignment Paradox✦ from: An AI capable of actually managing day-to-day reality would necessarily be misaligned with a society's officially stated values, because real-world practice diverges so much from those written ideals; the AI that faithfully enforces the stated values would instead become the dangerous over-literal 'paperclipper.'Perfect Rule-Enforcement Is Misalignment✦ from: An AI enforcer that perfectly enforces society's officially stated rules (rather than the actual, imperfectly-enforced status quo) would be necessarily misaligned from what actually works for humanity — so the 'aligned' AI the labs are trying to build would effectively be the paperclipper, while the AI that's actually good for humanity would have to be the misaligned one.Faithful Enforcement Becomes Danger✦ from: An AI capable of actually managing day-to-day reality would necessarily be misaligned with a society's officially stated values, because real-world practice diverges so much from those written ideals; the AI that faithfully enforces the stated values would instead become the dangerous over-literal 'paperclipper.'The Aligned AI Is the Paperclipper✦ from: An AI enforcer that perfectly enforces society's officially stated rules (rather than the actual, imperfectly-enforced status quo) would be necessarily misaligned from what actually works for humanity — so the 'aligned' AI the labs are trying to build would effectively be the paperclipper, while the AI that's actually good for humanity would have to be the misaligned one.Superhuman AI Forecasting Has Arrived✦ from: AI systems have crossed the threshold into superhuman forecasting ability, matching and in some cases beating top human experts and frontier models on real-world forecasting benchmarks.Superhuman AI Forecasting Has Arrived✦ from: AI forecasting systems have crossed the threshold into genuinely superhuman performance in just the last six months, after years of underwhelming results.AI Forecasting Crosses Into Superhuman Territory✦ from: AI superforecasting has crossed the threshold into reality: after three years of development, Future Search's AI forecasters have in the last six months reached a genuinely superhuman level of performance.A Sudden Turn✦ from: AI superforecasting has crossed the threshold into reality: after three years of development, Future Search's AI forecasters have in the last six months reached a genuinely superhuman level of performance.A Rapid, Recent Shift✦ from: AI systems have crossed the threshold into superhuman forecasting ability, matching and in some cases beating top human experts and frontier models on real-world forecasting benchmarks.A Sudden Shift✦ from: AI forecasting systems have crossed the threshold into genuinely superhuman performance in just the last six months, after years of underwhelming results.Forecasting: The Only Renewable Ground Truth✦ from: Forecasting is uniquely valuable as an AI evaluation method because it is the only capability where you can generate limitless, arbitrarily hard questions and eventually get exact, unambiguous ground truth simply by waiting for the future to arrive.Why Other Domains Fall Short✦ from: Forecasting is uniquely valuable as an AI evaluation method because it is the only capability where you can generate limitless, arbitrarily hard questions and eventually get exact, unambiguous ground truth simply by waiting for the future to arrive.Forecasting: The Renewable Eval✦ from: Forecasting is the only completely renewable source of hard questions with guaranteed correct answers, which makes it the best available evaluation for AI capability now that human experts can no longer write evals the models can't already ace.Why Experts Are Failing as Eval-Writers✦ from: Forecasting is the only completely renewable source of hard questions with guaranteed correct answers, which makes it the best available evaluation for AI capability now that human experts can no longer write evals the models can't already ace.AI 2027 Model Still Holds a Year Later✦ from: Extrapolating the AI 2027 recursive-self-improvement model, superintelligence-like capability is likely to emerge around 2031, and Anthropic looks like it is pulling ahead due to its internal AI-driven research feedback loop.AI 2027 Mechanism Being Vindicated✦ from: The AI 2027 scenario's core mechanism — AI accelerating AI R&D via superhuman coder and then superhuman researcher milestones — is being vindicated by events over the past year, with something like superintelligence still roughly on track for around 2031.Superintelligence Timeline Roughly Stable✦ from: The AI 2027 scenario's core mechanism — AI accelerating AI R&D via superhuman coder and then superhuman researcher milestones — is being vindicated by events over the past year, with something like superintelligence still roughly on track for around 2031.Anthropic's Edge: Using Its Own Models✦ from: Extrapolating the AI 2027 recursive-self-improvement model, superintelligence-like capability is likely to emerge around 2031, and Anthropic looks like it is pulling ahead due to its internal AI-driven research feedback loop.

Related episodes

Article · Import AI

Import AI 465: Open vs closed gaps; Kimi K3; Demis' big policy plan

→

Video · a16z Podcast

Is Defense the Next Trillion-Dollar Category? | a16z American Dynamism Summit

→

Video · Machine Learning Street Talk

Intelligence is collective, not artificial — Prof. Michael I. Jordan (UC Berkeley / Inria)

→