Thomas Ahle argues that reporting benchmark pass-rates (like '70-80% of tests correct') is misleading, since a program that fails even some tests is likely not truly correct overall. ✦ AI generated
Thomas Ahle · Machine Learning Street Talk · 2026-06-28 · original ↗
starts at this moment · 11:28
“If an AI can generate a chip design or a proof or a working program, how do you know it's actually correct?”
They always post this like oh the percentage of tests that it got right or like even like the Fable it was like oh yeah it got like 70 80% tests correct and it's like yeah but I talked with the people who made the benchmark and like yeah but it did it get any of them actually right? you know, like did it because it's if if the program only passes 70% of the tests, it's probably not right.
verbatim transcript · starts at 11:28
11:28they took 150 170 um programs. Some of them are really complicated like ffmpe is one of them. Um the AI just has to reimplement them uh without internet access. Yeah, basically when it came out they had like all of the LLMs got 0%. Because none of them like were able to pass all of the tests. But I've seen like people like I very rarely see that when people post benchmarks for this
11:53thing. They always post this like oh the percentage of tests that it got right or like even like the Fable it was like oh yeah it got like 70 80% tests correct and it's like yeah but I talked with the people who made the benchmark and like yeah but it did it get any of them actually right? you know, like did it because it's if if the program only
12:11passes 70% of the tests, it's probably not right. >> I know. And in a in a way, this is the thread that we were talking about before, you know, like for the last 60 years, going back to behaviorism, there's been this kind of um thread between uh structure and competence and prediction, right? So, essentially this this program bench is making the argument that you can learn the physomy
12:31of a program from its external behavior. So, you can learn the deep structure, the constraints and so on. And I suspect that that is not possible. >> Um, unless of course the LLM already knows about the source code because it's in its training data or whatever. But what do you think about that? >> Yeah, I think I think humans do it all the time actually. I think um there's
12:54the whole field of um kind of reverse engineering where people um yeah actually I listened to something with with with the FFM impact people where they were talking about all these codeexes that they that they put in there and they um like a lot of time they have no idea what they do like it's just like this obscure blob of code like they they probably don't even have
13:17access to the code they just have uh a couple of like example video examples that were encoded with this and maybe some people had shared some screenshots like with clips from the uh from the movie and then they had to go and and try and read this like encoded blob and being like what is it doing like how could we like write a program that decodes this thing and somehow people