Dan Schwarz says Future Search's AI forecasters, dismissed as unimpressive when built on Claude 2 in 2023, have in the last six months suddenly become good enough to claim superhuman forecasting ability. ✦ AI generated
Dan Schwarz · The Cognitive Revolution · 2026-07-07 · original ↗
starts at this moment · 61:15
“So the occasion for this conversation was this post that just came out from Scott Alexander. I know you guys have collaborated on a couple things over time. Uh but that really caught my eye because you know Scott is not uh I would say an uncritical AI hype man and yet the title of the post is the AI super forecasters are here. So tell us about it.”
future search was founded under the premise that eventually we would get to this superhuman level and yeah we claim that level uh has arrived. It's all very sudden. You know, we've been operating this company for about three years and it's really in the last six months that these things have started to really scare and impress us in their capabilities.
verbatim transcript · starts at 61:15
61:15>> Yeah. Um Scott was actually one of the first people to see the very first AI forecaster. Depends how you define forecaster. You know time series forecasting has existed for decades in AI and machine learning. It all depends how you define those terms. Uh but future search was started in uh 2023. We started building a forecaster before GBD4 turbo had even come out. We used claude 2. It's very early days. Uh Scott
61:38tried our forecaster then and was not super impressed and I don't think really anybody was back then. But of course future search was founded under the premise that eventually we would get to this superhuman level and yeah we claim that level uh has arrived. It's all very sudden. You know, we've been operating this company for about three years and it's really in the last six months that
61:56these things have started to really scare and impress us in their capabilities. I think the main thing to remark about forecasting is that it's it's a very important capability, but it is someone neglected. Described it as hyped, but really the whole field of forecasting is only about 15 years old. Prior to Tetlock, nobody was even studying accuracy at all. Nobody was recording predictions in a resolvable way and scoring them to give us even the
62:18most basic scientific evidence on what was going on. Um even before Tetlock, many of us were enthused by prediction markets, but that was a very human technology. Prediction markets are very big right now. Um as you said, Posh, I built the one that's currently running at Google. I think that's great. Uh but that's a way of trying to augment and coordinate human intelligence. Of course, we're more interested in
62:37artificial intelligence now. And so it's taken a little bit of time. You know, Midjourney was already very good at generating images. I don't know exactly how you would compare image generation, audio generation, you know, writing essays, doing coding, math, all the other capabilities that we normally track. At what point it became human, superhuman, it all depends on how you define it. Uh forecasting, if anything, is a bit late to the game, but now that
62:57it's here, it really changes the nature of what forecasting even is. You mentioned that in the last six months you have started to see things which are uh one step up or different from what you were seeing before. Can you can you elaborate? >> Yeah, I think uh the trouble with this and a lot of Scott's post was trying to cover the level of evidence because the