Forecasting is uniquely valuable as an AI evaluation method because it is the only capability where you can generate limitless, arbitrarily hard questions and eventually get exact, unambiguous ground truth simply by waiting for the future to arrive.

92:41some question about the future and basically an impossibly hard question, a question that even an AGI, an oracle, a god could never really say because of chaos theory. Imagine just trying to predict, you know, like cubic meter weather 3 weeks in the future. Like you'd never be able to do it. Um, but if you just wait, then you will see what that weather was in that cubic meter

93:03three weeks in the future. And so you basically have a completely limitless set of extremely hard basically impossible questions where you get exact ground truth. And there is no other email like this. There is if you want to say improve a coding harness, you just need to have more and more hard coding problems that are not in the training data for which you can say this is

93:23definitely the correct answer so that you can do some training on it. And that's hard. I think uh human experts, doctors, lawyers, engineers, financers, whoever who are trying to make evals to try to produce data for the frontier labs are finding that they are not smarter than the things being trained anymore. And so if you can produce something that has a correct answer, the model's already going to figure out that

93:43correct answer. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this. And again, this kind of is connected to forecasting as the, you know, the the kind of Elon Musk tweet the quips that forecasting is like the ultimate measure of intelligence. If you zoom out and think about it in one perspective, it

94:02is. Again, I think coding intelligence, AI, R&D intelligence, interpersonal intelligence are pretty darn important. I wouldn't say that forecasting is truly ultimate intelligence, but it is to some degree the ultimate eval. And I think this is something that Frontier Labs like Canon should be paying attention to. Do you have a sense for how much irreducible chaos there is? Like in the limit of super intelligence, how much visibility

94:31do we get into the future? >> Yeah, that is a great question. Uh David Mannheim actually just uh posted a tweet that has a graph that has a claim to estimate this that I was meaning to dig into because I don't know where those lines are coming from. Um I would say without evidence behind this my sense is very Ukowskian like I think there is a lot of detail in reality that is far