Using human-time-to-complete as the yardstick lets you place radically different models like GPT-2 and Opus on one unified, comparable axis of AI capability, which raw accuracy-based benchmarks cannot do because they saturate and get replaced.
David Rein explains that Meter's time-horizon metric was built to solve the problem that accuracy-based benchmarks can't be compared once models saturate them and labs move to harder ones — human completion time gives a single continuous axis from GPT-2 to today's frontier models. ✦ AI generated
David Rein · Machine Learning Street Talk · 2026-05-04 · original ↗
starts at this moment · 18:05
“For listeners who have only kind of seen the chart but they've not really read the paper, they don't understand it. Can you just go through it from a high level?”
I think about the key insight um of the time horizon's work being um to um yeah use this uh use this notion of human time to complete. So how long does the task take a human to do a human who kind of has a reasonable amount of expertise.
verbatim transcript · starts at 18:05
18:05coding questions or like write a short you know 20line Python program. Um, and so it's it's like it's very difficult to kind of uh you know at first at first blush to uh you know say like okay yeah like it's how you know how much harder is writing a Python program uh than you know finishing finishing the word uh in this in this paragraph. It's it's kind
18:25of hard to hard to think about that. And so I think about the key insight um of the time horizon's work being um to um yeah use this uh use this notion of human time to complete. So how long does the task take a human to do a human who kind of has a reasonable amount of expertise that uh such that you know uh you they would plausibly be doing the
18:46task in in in in their kind of either work or in their day-to-day. Um [snorts] uh and uh yeah we the idea was we can use this metric uh as uh uh yeah to to kind of represent the like difficulty of the task um in in in some sense and then um we can compare models um across a very wide range of capabilities you know all the way from from GPT2 um now up to
19:10uh you know opus 4.6 six. That's the kind of um that's the kind of high level motivation. And then um yeah, there there are a bunch of a bunch of details about how how exactly we we do this. So um you know, we start out and we uh we we create a bunch of tasks. Um that's the kind of first um f first step. So um you know, we created tasks that range
19:31from a few seconds to complete um all the way up to tasks that take like 10 or 15 hours uh for for humans to complete. We we hired a bunch of people and we did did a bunch of this ourselves. um of uh we call it baselining. Um so uh you know we we give people the tasks um in a kind of terminal environment that's uh uh uh
19:50you know designed to be uh uh almost identical to the environment that uh that agents uh have. So the same kinds of uh you know tools, the same um you know whe whether internet access is is turned on or off. Um and then we measure you know how long does it take them to to complete the task. Um yeah, as I as I mentioned, people are kind of selected
- ·Accuracy benchmarks saturate as models improve
- ·Saturated benchmarks get replaced with harder ones
- ·This breaks comparability across model generations
- ·Meter measures how long a task takes a skilled human
- ·Creates one continuous axis of capability
- ·Spans from GPT-2 to today's frontier models
- ·Same time-horizon axis spans early and frontier models
- ·Enables direct comparison across radically different eras
- ·Solves the benchmark-replacement problem structurally