Pangram Labs' AI-text detector can score writing as 0% human even when the author made substantial, sentence-by-sentence edits over nearly an hour, so its binary verdicts shouldn't be treated as proof of purely AI authorship.
Nathan reviews his own podcast intro essays as scored by Pangram Labs and finds a case where he substantially rewrote an AI draft over nearly an hour yet still got a 0% human score, arguing the tool's scores can't be treated as proof of purely AI authorship. ✦ AI generated
Nathan · The Cognitive Revolution · 2026-07-07 · original ↗
starts at this moment · 125:11
Overall Pangram is quite accurate. Um, and yet we have at least one example out of 400 or so essays where I think the zero score I would confidently assert is wrong and unfair and should not be the basis for like a pylon. you know, it would the the the crowd uh the digital mob would be like in the wrong for piling on somebody uh for passing off my um snowflake intro essay uh or you know for attacking it as being a AI slop output.
verbatim transcript · starts at 125:11
125:11if you if you called that one 25% AI, even maybe 75% AI, I would call that kind of acceptable. 75% would seem high given the fact that I basically did rewrite almost every section of the thing. But I did keep the structure. I did keep the general, but of course, the structure was derived from my examples, too. So it's not like there was it's not like I didn't have any hand in that. Um
125:36but yes, I think you know what can we say? Overall Pangram is quite accurate. Um, and yet we have at least one example out of 400 or so essays where I think the zero score I would confidently assert is wrong and unfair and should not be the basis for like a pylon. you know, it would the the the crowd uh the digital mob would be like in the wrong for
126:06piling on somebody uh for passing off my um snowflake intro essay uh or you know for attacking it as being a AI slop output. I think I can show this edit history and everybody should agree that like yeah, you put in an hour, you basically rewrote almost every section and somehow Pangram still gave you a zero from this. I would say you should not you cannot convict in a you know
126:34reasonable doubt system purely based on this sort of thing and yet at the same time you can pretty much trust the panagram signal as a consu so as a consumer I think you can trust it as a judge I think you should be more cautious >> I wonder to what extent uh people reading AI slop get trained on AI slop and start writing like AI slop and so
127:05>> yeah all this stuff's going to blur. I mean >> yeah so we we are we are very malible like the way we the way we speak and express ourselves is very malible. So um I can I I could definitely see for example uh countries where English is a second language and they pick up mostly off the web or chat totally will totally like you know they