Building a coherent 'world model' out of thousands of interlinked AI forecasts risks the same kind of correlated reasoning failure that produced the 2008 financial crisis, and Dan's own attempt to forecast the Claude Fable export-control situation demonstrated exactly this failure mode.
Dan recounts building a detailed multi-scenario forecast of the Claude Fable export-control decision and getting a key assumption wrong across every scenario, illustrating how correlated errors can silently corrupt an otherwise sophisticated forecasting model. ✦ AI generated
Dan Schwarz · The Cognitive Revolution · 2026-07-07 · original ↗
starts at this moment · 113:38
“can we protect ourselves against that in any way?”
one thing came out which is basically every forecast and every scenario I had thought that access would come to Americans first and then foreigners at some later point in the future and that was wrong uh when it came out last week it came back for everybody so clearly there was some weight in one of my scenarios that was wrong.
verbatim transcript · starts at 113:38
113:38the outcomes and I, you know, talked about it with claude code a lot. one thing came out which is basically every forecast and every scenario I had thought that access would come to Americans first and then foreigners at some later point in the future >> and that was wrong uh when it came out last week it came back for everybody so clearly there was some weight in one of
113:57my scenarios that was wrong but I had a basically like kind of a correlated failure in there somewhere I still haven't completely understood where my reasoning was wrong it's also possible I just got really unlucky and the outcome we're in was just extremely unlikely this n equals one you can never know if any one forecast is great. That's one of the hard things about it. But I think I
114:14systematically got it wrong by having a bunch of correlated reasoning failures across my various scenarios. So this definitely does happen. Uh Metaculus has a system like this. Um in the years since I was the CTO there, they have built an actual causal graph platform and product. So you can go to the Metacula site and click around and you'll find it there. Um I think the field still generally believes that
114:38things like this will work but nobody has actually made a good one before and uh I tried my best over basically like you know 12 to 16 hours of the fable situation. I think I made a pretty good model. I think I did I was close to having a very accurate forecast but I didn't quite get it. I don't think um I don't think those metaculous models on
114:56their website right now are so amazing. But I do fundamentally believe in the approach. As you're saying, Nathan, this has been tried for a long time. When I was the CTO of Metaculus, honestly, it was it was kind of the dream. It was the holy grail. Can we tie all of these forecasts together into some sort of causal graph? Um, and I think what I can say is that AI makes this tractable.
115:17There was just no way that that was going to work with a bunch of human economists looking at Freddy Mack or Fanny May. I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear. Whether AI will make this work in general feels nearly guaranteed and I don't think future search is the only org that is working on this right now.