Audio · 2026-06-26 · 36m · 6 moments

Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI Research Scientist Noam Brown

When a new AI model drops, it’s judged based on a static benchmark grid that doesn’t account for how long the model is allowed to think. How then should we measure a model’s true capability? OpenAI research scientist Noam Brown returns to talk with Sarah Guo about his latest essay on why the AI industry’s traditional benchmark grids are broken, and how large-scale test-time compute is fundamentally changing how models are evaluated. Noam explains how, if properly scaffolded, today’s models can r ✦ AI generated

Apple Podcasts ↗

GUEST Noam Brown HOST Sarah Guo HOST Elad Gil

timeline · colored by role

Claim◆

The capability of a model is now a function of how much test-time compute you invest in it, not a fixed property, and existing evaluation frameworks (preparedness frameworks, responsible scaling policies) do not account for this.

Noam argues that traditional safety and capability evaluations treat model performance as a static number, but with modern models capable of sustained reasoning over weeks, capability scales with compute budget — a question the current frameworks ignore.

transcript

Noam Brown: The preparedness frameworks and responsible scaling policies, they don't really account for the amount of test time compute. They just say, okay, well, what's the capability of the model? The problem is we're in a world now where the capability of the model is a function of how much money you put into it, basically. If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. Give it a budget of $10 million, it could do even more. And so at what budget should you evaluate these models? The policies that exist today don't really address that question.

extends · 1

Claim◆

Benchmark grids are misleading because they do not control for the amount of test-time compute used, making models that are more efficient thinkers look less impressive on paper than they are in practice.

Noam explains that o5.5 looked only modestly better than o5.4 on standard benchmark grids, but once you account for thinking time, o5.5 is a substantial jump because it is far more efficient with its reasoning.

transcript

Noam Brown: I think the reason why it doesn't show up as so much better on the benchmarks is because the benchmarks are being presented, the benchmark results are being presented in the wrong way. They're not controlling for the amount of test time compute that is being used on that benchmark question. It turned out that o5.5 is just much more efficient with its thinking. If you run it at max settings, o5.4 is thinking for a lot longer. It takes longer to get back a response than o5.5. And once you control for the amount of thinking time, actually you can see that o5.5 is a substantial jump over o5.4.

provides context · 1

Claim

Modern models can think productively for weeks before performance plateaus on many benchmarks, so evaluating at the plateau point is no longer practical — we need to evaluate with a fixed budget (tokens, cost, or time) or plot performance as a function of compute.

Noam explains that unlike GPT-3 era models, which plateaued quickly, today's models like o5.5 can sustain productive reasoning for weeks. The plateau point is too far out to test, so evaluation must switch to plotting performance against a compute budget.

transcript

Noam Brown: The thing is, the point at which it plateaus is actually really far out these days. I mean, if it's true in GPT-3 land back in 2022, the models couldn't really think productively for that long. And so you could just run them until they plateau. It's not that far away. But what we're seeing today with the modern models is that o5.5 and other models can think for, if you scaffold them reasonably well, can think for weeks even, before having performance plateau on some of these benchmarks. And so the point at which they plateau is simply too far out to reasonably test. ... And so my claim is the proper way to evaluate the models now is you either have some kind of budget for the benchmark, whether it's tokens or cost or time or whatever, or you plot the performance as a function of the amount of test time compute that's going into the model.

explains mechanism · 2

Claim

Nobody actually knows the ceiling of capabilities for current models because the model release cycle (every 2-3 months) is faster than the time needed to fully explore what a model can do with long-running inference (weeks to months).

Noam points out that models are released every few months, but running them to their limits on sustained tasks takes months — so no one truly knows what the current generation can do at its ceiling.

transcript

Noam Brown: The model release cycle is, look, we're releasing new models like every two or three months at this point. And so a model comes out, it takes two or three months to push it to its limits, and then you have another model come out. And so nobody actually knows what the ceiling of capabilities are for these models because nobody's actually run them for long enough to really tell.

explains mechanism · 3

Example

The Erdos unit distance conjecture was disproven by an internal OpenAI model at a very low compute budget, and the same result could have been obtained from o5.5 with a sufficiently expensive general-purpose scaffold — but nobody had explored that because the cost drops 10-100x with each model release cycle.

Noam describes how an internal OpenAI model disproved the Erdos unit distance conjecture at very low cost. The same result could have been achieved from o5.5 using a general-purpose scaffold costing $1,000–$100,000, but nobody had tried because rapid model improvement makes such expensive runs feel not worth it.

transcript

Noam Brown: After we announced the results, a bunch of people found that you could get the answer out of o5.5 as well. If now, it's not as simple as just asking o5.5, hey, here's the unit distance conjecture. What's the disproof? You had to scaffold it a bit. ... What this means is you could, in principle, ask o5.5 to, as a general purpose scaffold, list a bunch of different strategies, and then for each strategy, tell to investigate that strategy. And then it would probably be able to arrive at the disproof with a general purpose scaffold. Now, that scaffold would be very expensive. I mean, it would probably cost, I just ballpark like $1,000 to $100,000. But it would be possible, and it would have been possible for somebody to disprove the Erdos unit distance conjecture before we did using a general purpose model. And nobody had explored sufficiently what happens if I put $100,000 worth of compute into o5.5, what could it do?

gives example · 3

Claim

Test-time compute does not improve model performance uniformly across all tasks — it helps on some tasks (e.g., Sudoku, mathematical reasoning) and not on others (e.g., factual retrieval), so models are not at the point where enough compute alone replaces human researchers.

Noam explains that more thinking time helps models on reasoning tasks like Sudoku (brute-force solvable) but not on simple factual recall. Current models lack research taste and cannot fully replace the research cycle, though this may change over time.

transcript

Noam Brown: There are some benchmarks where the models will just not improve if they have more inference budget. So I think a lot of factual retrieval kind of questions fall into this category... if you give them a week, they're not suddenly going to do better at remembering dates. ... On the other extreme, there are benchmarks where they kind of obviously will keep improving without limit with more test time compute. So the example I like to point to is Sudoku. ... The models are not at the level where if you just give them enough test time compute, they will be able to do all of our jobs, just because, yeah, there's some benchmarks where they will not improve. ... One thing I see for research in particular is they don't have very good research taste right now.

extends · 3

Highlight slides

Capability Is a Function of Compute Budget✦ from: The capability of a model is now a function of how much test-time compute you invest in it, not a fixed property, and existing evaluation frameworks (preparedness frameworks, responsible scaling policies) do not account for this.The Unaddressed Question✦ from: The capability of a model is now a function of how much test-time compute you invest in it, not a fixed property, and existing evaluation frameworks (preparedness frameworks, responsible scaling policies) do not account for this.Benchmark Grids Are Misleading✦ from: Benchmark grids are misleading because they do not control for the amount of test-time compute used, making models that are more efficient thinkers look less impressive on paper than they are in practice.o5.5 vs o5.4: Efficiency Reveals the Real Jump✦ from: Benchmark grids are misleading because they do not control for the amount of test-time compute used, making models that are more efficient thinkers look less impressive on paper than they are in practice.

Related episodes

Article · Import AI

Import AI 464: Fables writes GPU kernels; AI automation; and analog computation

→

Article · Latent Space

[AINews] Thinky's Inkling: 975B-A41B multimodal, new best American Apache 2.0 open model (with Inkling-Small, 276B-A12B)

→

Article · Import AI

Import AI 465: Open vs closed gaps; Kimi K3; Demis' big policy plan

→