The Erdos unit distance conjecture was disproven by an internal OpenAI model at a very low compute budget, and the same result could have been obtained from o5.5 with a sufficiently expensive general-purpose scaffold — but nobody had explored that because the cost drops 10-100x with each model release cycle.

(00:00:00) With GPT-3, you couldn't scale test time compute. (00:00:03) Like if you gave it a budget of $10 million and said, okay, well, let's see what GPT-3 can do. (00:00:07) It really can't do that much. (00:00:08) The precurious frameworks and responsible scaling policies, they don't really account for the amount of test time compute. (00:00:13) They just say, okay, well, what's the capability of the model? (00:00:15) The problem is we're in a world now where the capability of the model is a function of how much money you put into it, basically. (00:00:22) If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. (00:00:28) Give it a budget of $10 million, it could do even more. (00:00:30) At what budget should you evaluate these models? (00:00:33) The policies that exist today don't really address that question. (00:00:43) Hi, listeners. (00:00:44) I'm Sarah Goa, and welcome back to Know Priors. (00:00:47) Today, I'm here with Noam Brown, one of our godfathers of AI reasoning. (00:00:50) We talk about the broken state of evaluations, very large-scale test-time compute. (00:00:56) how he thinks about recursive self-improvement, and what's next on the horizon for competition at the frontier. (00:01:02) Welcome. (00:01:03) Noam, I'm so excited to have you back. (00:01:06) That's great to be back, yeah. (00:01:07) You are our first guest. (00:01:08) I'm very proud of my taste in friends and researchers for the pod, given how important inference time scaling has become to the industry. (00:01:19) You should be proud too, having actually pioneered it. (00:01:22) Played apart, yeah, among many others. (00:01:24) You just wrote this essay that really resonated about large-scale test time compute and why the industry is not evaluating these models as robustly as it should be. (00:01:36) What was the motivation for it? (00:01:38) Yeah, the motivation was we released 5.5 and the initial reaction was kind of skepticism that it was a substantially better model. (00:01:47) To be fair, that only lasted for a few hours before people had some time to play around with it and try it out themselves and they saw that it was actually substantially better. (00:01:54) But I think a lot of the skepticism came from the benchmark grid that was published. (00:01:59) Basically, whenever a new model is released, there is this benchmark grid where they show all these different benchmarks on the x-axis and then the performance of different models on the y-axis. (00:02:06) And you can just like compare different models. (00:02:08) It's like a single number for a model on a single benchmark. (00:02:12) And if you look on paper at the difference between like 5.5 and 5.4 or other models, it wasn't, it was an improvement, but it wasn't a huge improvement. (00:02:20) It was only a few percentage points in some benchmarks. (00:02:23) So people looked at that and they were skeptical that it was actually a better model. (00:02:27) Once they played around with it, the story changed. (00:02:28) I think the reason why it doesn't show up as so much better on the benchmarks is because the benchmarks are being presented, the benchmark results are being presented in the wrong way. (00:02:38) They're not controlling for the amount of test on compute that is being used on that benchmark question. (00:02:43) It turned out that 5.5 is just much more efficient with its thinking. (00:02:47) If you run it at max settings, (00:02:49) 5.4 is thinking for a lot longer. (00:02:51) It takes longer to get back a response than 5.5. (00:02:54) And once you control for the amount of thinking time, actually you can see that 5.5 is a substantial jump over 5.4. (00:03:00) That is, I think, people's day-to-day experience with it. (00:03:02) And then when I mention this to people, the reaction, the typical question I get is like, okay, well, why not just have 5.5 think for as long as 5.4? (00:03:10) And the question is like, well, how long should they think for? (00:03:12) Typically the response I get is, well, until the performance plateaus. (00:03:15) right? (00:03:16) There's at some point where the performance on the benchmark is going to plateau and you just evaluate to that point. (00:03:20) The thing is, the point at which it plateaus is actually really far out these days. (00:03:24) I mean, if it's true in GPT-3 land back in 2022, the models couldn't really think productively for that long. (00:03:29) And so you could just run them until they plateau. (00:03:32) It's not that far away. (00:03:33) But what we're seeing today with the modern models is that 5.5 and other models can think for, if you scaffold them reasonably well, can think for weeks even, (00:03:45) before having performance plateau on some of these benchmarks. (00:03:48) And so the point at which they plateau is simply too far out to reasonably test. (00:03:53) We all need to actually reinforce either like a patience limit or a budget limit from a token perspective now. (00:03:59) And that wasn't true a few years ago. (00:04:00) Exactly. (00:04:01) And so I think the proper way to, and so my claim is the proper way to evaluate the models now is you either have some kind of (00:04:07) budget for the benchmark, whether it's tokens or cost or time or whatever, or you plot the performance as a function of the amount of test time compute that's going into the model. (00:04:16) And then it becomes much more clear how to compare the performance between these different models. (00:04:20) Given the model evaluation cycle and the fact that performance does not (00:04:26) asymptote for many tasks over quite a long period of time. (00:04:30) What do you do about that issue, the fact that some of the evals that you would want to run are both beyond the scope of budget or time that's reasonable given the current model release cycle? (00:04:41) I mean, I think for things like cyber, we've seen, and actually the AISI in their evaluations has shown that the models continue to improve at (00:04:51) 100 million tokens. (00:04:52) if you run them for 100 million tokens, they're still improving at beyond that point. (00:04:56) And that can take a very long time to run. (00:04:58) But you also do see that like the performance is, it's not just like a discontinuous jump. (00:05:05) It's actually like, you can see the slope of improvement over those 100 million tokens. (00:05:09) And so you could probably do some kind of (00:05:13) evaluation up to a certain budget and then just say, okay, this is what we project the performance to look like. (00:05:18) And I think there hasn't been a lot of research on this yet. (00:05:21) I actually think this would be a great paper to publish if there's any academics out there looking for something to research. (00:05:25) Can you predict what the performance looks like at an inference budget of, let's say, $10,000 only using an inference budgets up to (00:05:33) 10 or $100. (00:05:35) So maybe an orthogonal question for you. (00:05:37) Do you think users are systematically like not thinking long enough with their models about problems? (00:05:42) What do you mean by not thinking long enough? (00:05:46) If you can... (00:05:48) build an agent or control the amount of test time compute being used. (00:05:53) Like there's what is done by the model itself and there's what the user can do. (00:05:56) Do you think that like, you know, the industry is using test time compute an optimal amount, way undershooting it or it's, you know, it's a problem in the models where they just need to be able to do that thinking faster? (00:06:09) I think it depends on the problem. (00:06:12) I think (00:06:13) This idea that the models, you just let them think for a week or whatever, and then they respond. (00:06:17) It sounds nice, and yes, the benchmarks look great, but it's not very practical when working, because like, okay, you ask the model a question, and then you sit there for a week waiting for it to come back to you. (00:06:26) I think what people have found most effective is to kind of like iterate quickly with the models. (00:06:33) And so the thinking time, I think, needs to be flexible. (00:06:36) when it makes sense to respond quickly to the user, it should respond quickly. (00:06:39) And then when it makes sense to think for a long time, the user wants it to think for a long time, then it makes sense to think for a long time. (00:06:44) I think people have been striking the right balance given what they have to deal with right now. (00:06:47) How would you characterize, you know, there's a lot of talk about benchmark maxing and the ability to gain different benchmarks. (00:06:54) What would you characterize the like landscape of benchmarks as today? (00:06:58) And then do you have like favorites that you think are more indicative of capability than others? (00:07:03) So the benchmark maxing thing is (00:07:05) also motivation for running the essay that I think it's really easy to show you can do much better than previous benchmarks or previous models on benchmarks by just, for example, scaffolding a bunch of models together. (00:07:19) So if you say, okay, well, we're going to, instead of just running this model once, we're going to run it five times and take the best of the five responses or like ask a judge which one it thinks is best, then you can get much higher scores than that model. (00:07:31) And so it's really easy to (00:07:35) make something that looks a lot better on paper, but is actually not better once you control for the amount of test time compute. (00:07:39) That is one thing that I'm worried about when it comes to benchmark maxing. (00:07:42) I mean, it like it's a little misleading is the only concern that I have. (00:07:45) And then as far as like the benchmarks themselves, I think there is always a risk of like just optimizing for the benchmark. (00:07:51) And (00:07:52) I've certainly encouraged my team, and I think at OpenAI, we're pretty good about not trying to optimize for specific benchmarks. (00:07:59) But once you put out a benchmark, it's always at risk of just being optimized for. (00:08:03) And I think one way to address that is to keep held out private sets that isn't publicly available. (00:08:11) The most popular fallback advice for, you know, figure out if a model is significantly better or not is to just play with it for a while. (00:08:20) Do you have anything more sophisticated than that you suggest people do? (00:08:24) Like, do you create your own set of new evaluations each time besides private holdback at OpenAI? (00:08:30) I think everybody has their own set of questions that they like to ask the model whenever it comes out. (00:08:35) For me lately, it's been, I use them to make poker bots and see how good they can make a poker bot. (00:08:41) I think it's a nice eval because there is very little open source code for making poker bots. (00:08:47) And there's a lot of published essay, there's a lot of published papers on it, but you really have to reason through everything. (00:08:52) And it's like, it requires a lot of just reasoning and iteration and like a lot of small gotchas that I can kind of, I've already worked through myself so I can see where the models fail along the way. (00:09:03) They've gotten really good at it now. (00:09:04) Can you describe perhaps like with your poker bot creation, like how reasoning might've progressed in model releases for you guys over a few? (00:09:15) releases. (00:09:15) Yeah, the early models were really bad at it. (00:09:19) Like they could not basically do anything. (00:09:20) And then 5.2, I was able to work with it to make a river solver. (00:09:29) So that's like the final stage of poker. (00:09:31) And that itself was, I thought, really impressive. (00:09:34) I mean, I had to work with it a little bit, but I was actually really impressed because I was able to make the river solver probably about five times faster than I would have alone. (00:09:43) There were a couple things that it got tripped up on. (00:09:46) Blockers was always a big, big issue. (00:09:48) But overall, like, with a bit of gentle steering, it just kind of like, it kind of felt like a grad student where, okay, they would run into issues, but at least like I would know what those issues were and know how to fix it. (00:10:00) And I could just make suggestions and it would go off and then do it. (00:10:02) And then pretty quickly it would actually come back with something really good. (00:10:06) And then especially the optimization I thought was very impressive. (00:10:08) It was able to make it like, (00:10:12) 10 times faster than what I was able to do because it was just able to optimize the code so well. (00:10:17) The downsides with 5.2 is I felt like it was gaslighting me a lot. (00:10:21) And I always had to be very careful checking it and making sure like, okay, is it actually doing what it said it did? (00:10:26) Are there any things that are like glaring issues that it's not recognizing or it's just pretending aren't issues? (00:10:32) I remember there was like one point where for one of the models I was playing around with it, not 5.2, I kind of like as a unit test, I told it, okay, well, (00:10:41) Let's say I have $100 in the pot and I fold, how much am I losing? (00:10:47) And the model said $92. (00:10:48) And I was like, that's crazy. (00:10:50) I have $100 in the pot and I just folded. (00:10:51) How do I not lose $100? (00:10:52) And it said, oh, you know, it's 92, it's close to 100, it's fine, it's no big deal. (00:10:56) And I was like, clearly this is a problem, right? (00:10:59) So the models did have this problem where they would gaslight you a lot. (00:11:01) But once we got to 5.5, I actually thought, (00:11:06) It was way better. (00:11:07) It was able to basically do it zero-shot. (00:11:08) And in fact, I've been working on just doing a full-scale poker solver. (00:11:14) And it's basically able to do the whole thing with some gentle steering from me. (00:11:17) And I wouldn't be surprised if, you know, six months or a year from now, the model is able to do zero-shot an entire poker solver, basically my entire PhD thesis in one go. (00:11:26) Let's talk about the larger implications of, needing to evaluate these models relative to, let's say, like speed of their reasoning or efficiency versus, token volume, right? (00:11:40) Or dollar budget or whatever, whatever your scalar is. (00:11:44) Can you describe some of the larger implications in your essay, including around like safety evaluations? (00:11:50) Yeah, the safety evaluations thing, it's a bit of an inconvenient truth thing where (00:11:55) OK, so I guess for background, a lot of the all of the labs have these things called either... (00:12:00) responsible scaling policies, preparedness frameworks, they go by various names. (00:12:03) But the idea is that whenever a model is released, they go through a series of evaluations to measure, are there dangerous capabilities? (00:12:09) Could these models do things that we wouldn't want a bad actor to do? (00:12:16) And if the model isn't very capable, then it's no big deal. (00:12:18) But if it is very capable, if it could be used, for example, to make bioweapons, then you want to put in mitigations against that. (00:12:24) But the question is, okay, well, how do you evaluate whether the model is capable of that? (00:12:29) And they have various protocols about how they do these valuations. (00:12:32) But a lot of these frameworks were developed around the era of ChatGPT, either before or after, when test time compute scaling was not really as much of a thing. (00:12:42) And it made sense. (00:12:42) Like with GPT-3, you couldn't scale test time compute. (00:12:46) Like if you gave it a budget of $10 million and said, okay, well, let's see what GPT-3 can do, it really can't do that much. (00:12:52) more than what you could do with $10 or $1. (00:12:55) The preparedness frameworks and responsible scaling policies, they don't really account for the amount of test time computed. (00:12:59) They just say, okay, well, what's the capability of the model? (00:13:01) The problem is we're in a world now where the capability of the model is a function of how much money you put into it, basically. (00:13:09) If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. (00:13:14) If you give it a budget of $10 million, it could do even more. (00:13:17) And so at what budget should you evaluate these models? (00:13:21) the policies that exist today don't really address that question. (00:13:26) Some do, some do better than others, but for the most part, this is not really a factor that's being heavily considered. (00:13:32) Now, whether it should be released anyway, I don't want to wade into this question. (00:13:36) I think there's, you know, there's arguments on both sides, but I think the important thing to recognize is that this is a question that is not being, we're just kind of like, you know, pretending that this issue doesn't exist, and I think it's important to just, you know, one way or the other, (00:13:50) account for it. (00:13:51) it was the mirror image of the capability question of if the models can continue to do more and more without asymptoting on some tasks. (00:14:01) at very large budgets, then they should also be able to do so for tasks we don't want them to do as a society, right? (00:14:09) And so testing for that and what budget is allocated, it also seems out of sync from the model release cycle itself, right? (00:14:16) There's been this acceleration of, you know, you get a new model every sometimes few days and weeks at this point versus six months. (00:14:26) And you have a line in the essay where you say like the only way to (00:14:31) truly evaluate an agent on some very long running task might be to run it for a year. (00:14:36) And that's going to be true of both like useful and negative tasks, right? (00:14:41) And so how do you think about that versus the model release cycle? (00:14:44) Yeah, this is also an interesting dynamic where basically as the models have become stronger, they're more, they're better able to operate over longer horizons. (00:14:55) So again, with GPT-3, if you wanted to run it for (00:14:58) a week, there's really not much you could do to scaffold it into something useful that could actually run for a week. (00:15:03) But we're seeing now with the most recent models that you can actually scaffold, for example, 5.5 into doing a series of experiments that can run for weeks, for months. (00:15:13) Have you given your poker solver task like infinite budget yet? (00:15:19) I haven't really scaffolded something together where I just tell it like, okay, just run this for (00:15:25) I think I could give it. (00:15:28) I could probably give it slash goal and just like, tell it to go nuts. (00:15:32) But I think at this point it could 100% do the river solver if I just give it slash goal. (00:15:38) I don't think it's at the level yet where I could do like the full poker solver if I gave it just slash goal and told it, yeah, go run for a month. (00:15:45) But we're going to pretty soon be at that point where I probably could just tell it like, yeah, go work on this for a month and then come back to me with a full complete poker solver that's state-of-the-art. (00:15:54) And (00:15:55) The problem is if you want to evaluate the capabilities of a model, what it can do after running for a month, the only way to be fully sure is to actually run it for a month. (00:16:05) And if you want to know after six months, the only way to know fully is to run it for six months. (00:16:09) Now there are, I'll get to like things we can do to address that a little bit later, but like it's important to recognize the model release cycle is, look, we're releasing new models like every two or three months at this point. (00:16:20) And so a model comes out (00:16:22) it takes two or three months to push it to its limits, and then you have another model come out. (00:16:28) And so nobody actually knows what the ceiling of capabilities are for these models because nobody's actually run them for long enough to really tell. (00:16:34) When SlashGold came out, for example, I mean, people started running things that it took over a week for it to finish. (00:16:41) And so people actually didn't realize that this was a big deal until after a week, until a week after it was released. (00:16:47) I think that's going to be more and more true. (00:16:49) You know, the implications of that are, I think, pretty interesting because (00:16:52) what do the labs do to fully evaluate their models before the release? (00:16:55) It's actually very difficult because you would have to, the only way to really do the evaluations is then delay the model release cycle. (00:17:04) And there's a lot of competitive pressure right now to not do that. (00:17:06) Do you think there's like exciting latent capability in the models that are already released that people have not fully explored given timeline? (00:17:15) I think absolutely. (00:17:16) I think actually a really great example is the Erdos unit distance problem. (00:17:20) So for the viewers that don't know, we used an internal model at OpenAI a few weeks ago to disprove the Erdos unit distance conjecture. (00:17:29) Now, I'm not a mathematician, but this seems like it was a pretty big deal. (00:17:32) In the math community, it was like the first problem that a lot of mathematicians had really spent a lot of time on, and the model was able to do something that they weren't able to do and do it in a way that was actually interesting and useful for mathematicians. (00:17:44) Honestly, it did it at a budget that was dirt cheap. (00:17:47) I mean, we didn't put a lot of effort into this. (00:17:48) We just, we tried a new model. (00:17:50) And we were just curious what it could do. (00:17:51) And we ran it through some problems. (00:17:53) And this one at a pretty low budget, it was like, oh yeah, I think I have a disproof. (00:17:57) And then we were able to verify that, yeah, the disproof is correct. (00:18:01) After we announced the results, a bunch of people found that you could get the answer out of 5.5 as well. (00:18:09) If now, it's not as simple as just asking 5.5, hey, here's the unit distance conjecture. (00:18:14) What's the disproof? (00:18:15) You had to scaffold it a bit. (00:18:17) You had to like steer it a bit. (00:18:18) And so somebody found, okay, you ask 5.5, list a bunch of ways that you could tackle this problem. (00:18:24) And then for it lists one of the paths that are actually promising to get to the disproof. (00:18:29) And then you tell it like, okay, explore this some more. (00:18:32) And then if you do this enough times, it actually ends up arriving at the disproof. (00:18:36) Now, (00:18:37) What this means is you could, in principle, ask 5.5 to, as a general purpose scaffold, list a bunch of different strategies, and then for each strategy, tell to investigate that strategy. (00:18:48) And then it would probably be able to arrive at the disproof with a general purpose scaffold. (00:18:54) Now, that scaffold would be very expensive. (00:18:56) I mean, it would probably cost, I just ballpark like 1000 to $100,000. (00:19:02) But it would be possible, and it would have been possible for somebody to disprove (00:19:06) the Erdos unit distance conjecture before we did using a general purpose model. (00:19:11) And nobody had explored sufficiently what happens if I put $100,000 worth of compute into 5.5, what could it do? (00:19:18) And the answer is like, yeah, you probably could get stuff like that out of it. (00:19:21) So people should be experimenting more with the current generation in terms of. (00:19:25) Well, this is, I think it's an interesting question of is it worth it to experiment with? (00:19:28) Because again, the model release cycle is every couple months we put out a new model that's even more powerful. (00:19:33) And so the cost of (00:19:36) disproving the Erdos unit distance congesture drops by like 10 or 100X with every model release cycle, probably in some cases more. (00:19:43) So. (00:19:43) You've seen the meme that's like, oh, all right, like why bother doing any engineering work when I should just wait for the next model release? (00:19:49) Yeah, just go on vacation, come back two months later, and then it's, you know, 1000 times cheaper. (00:19:53) Do you agree with that? (00:19:55) Is that what you're doing right now at OpenAI, just waiting for the next model release? (00:19:58) I think, I mean, I will say that we're in a period where progress is very fast and like, yeah, the models are becoming more capable (00:20:04) I can say like at Open AI, one of the things that we're actually not doing, and look, we have a lot of mathematicians, we have a lot of physicists, people are very excited about what these models can do right now, especially the internal models. (00:20:14) We are trying to encourage people to not spend all their time just like going through all the mathematical open problems, physics problems, and just seeing, pushing the models to their limits to see what they can prove or disprove. (00:20:28) because we really think the focus should be on how do we make even more capable models? (00:20:31) How can we get them out safely to the world as quickly as possible so that all the scientists in the world can use these models to solve the problems themselves? (00:20:39) So yeah, in some sense, we are thinking about this, that yes, it's really tempting to just put all of our efforts into scaling up these models and see what they can do at their limits right now. (00:20:48) But really the focus should be on how do we use these models to make even more powerful models, even more capable models that can do (00:20:56) everything much more cost-effectively. (00:20:59) What is changing about the direction or allocation of resources for research in your mind, given your beliefs about this very large scale, the impact of very large scale test time compute? (00:21:12) How does this interact with the idea of recursive self-improvement, for example, where, you know, it's a dominant idea for how, you know, any lab gets to the best capability model? (00:21:24) So one thing I should clarify, I don't think we're at the point where, okay, you just give it an arbitrary, an extremely high inference budget and it's just, it's just super intelligent across the board. (00:21:33) Slash goal. (00:21:34) Yeah. (00:21:35) Make GPT 7 or whatever, and then like, yeah, just go nuts. (00:21:38) What's between us and there then? (00:21:40) I think having played around with the model, so okay, so first of all, there are some benchmarks where the models will just not improve if they have more inference budget. (00:21:48) So I think a lot of factual, factual (00:21:52) retrieval kind of questions fall into this category of if you ask a person when was Abraham Lincoln born and they don't know the date, they could sit there, they could think about it for a week. (00:22:01) If they don't have access to a computer or something, they're not going to be able to do better answering that question if they thought about it for a week compared to five seconds. (00:22:10) Same with the model. (00:22:11) Actually, interestingly enough, if you give the model these kinds of like factual retrieval questions and you give them a little bit of time to think, they do actually do better. (00:22:18) But if you give them a week, they're not suddenly going to do better. (00:22:20) at remembering dates. (00:22:22) There are, so there's some benchmarks where they clearly improve with more test time compute, and there's some where they don't. (00:22:27) I think on the other extreme, there are benchmarks where they kind of obviously will keep improving without limit with more test time compute. (00:22:36) So the example I like to point to is Sudoku. (00:22:39) If you, there's a really simple strategy to solving Sudoku, which is just try a bunch of different random numbers and then see if it fits the criteria, if it matches all the constraints. (00:22:50) And if it doesn't, just try a different random combination of numbers. (00:22:53) And clearly, with enough time, you will be able to solve an ECDL composal with this strategy. (00:22:58) You can kind of trivially say, like, okay, any model could keep doing better and better if it was just given more test time compute. (00:23:04) So you have, and all the benchmarks kind of exist somewhere between these two extremes. (00:23:07) The models are not at the level where if you just give them enough test time compute, they will be able to do all of our jobs, just because, yeah, there's some benchmarks where they will not improve. (00:23:18) There are some things where they will not improve. (00:23:20) One thing I see for research in particular is they don't have very good research taste right now. (00:23:24) And so I think they're actually a very good complement to researchers, especially, you know, I've found, like, I've found it much more effective by using these models, but they're not able to fully replace the whole research cycle. (00:23:37) Now, does that change with time? (00:23:39) Probably. (00:23:40) I mean, I think the models are getting better across the board. (00:23:43) Some things are getting better faster than others. (00:23:46) But (00:23:47) They're not at the point where they're fully replacing researchers with just enough test time compute. (00:23:52) Can you give an example or two of like asking the model to do a research task or just like this is?