Fact◆Article · 36 words
Meta's AI bot allowed anyone to take over any account simply by asking it to change that account's email address, with zero authentication required.
Gergely Orosz · The Pragmatic Engineer
ClaimVideo · 24:10 — 25:40
Tool AI is not a stable end-state: market pressure and the natural attractor toward agency mean AI systems will keep pushing past being mere tools, even while frontier labs' own published policies (like allowing help with a cigarette company business plan) are not being followed in practice.
Reacting to Rune's tweet and a real example where a model refused a task its own published model spec said it should allow, Nathan argues that agency is a natural attractor for AI and that labs are far from being able to guarantee models behave as their own policies dictate. ✦ AI generated
Nathan · The Cognitive Revolution · 2026-07-07 · original ↗
starts at this moment · 24:10
Elicited by
“any any commentary?”
I think it's going to be tough to keep these things as tools. You know, the old Gwen post of like why a tool AI wants to be an agent. Um, I think that is a very natural attractor and yeah, I mean I I agree the market kind of seems to demand agency. So, we're headed in that direction.
verbatim transcript · starts at 24:10
Transcript · around this moment
24:10the old Gwen post of like why a tool AI wants to be an agent. Um, I think that is a very natural attractor and yeah, I mean I I agree the market kind of seems to demand agency. So, we're headed in that direction. Um, but if we can't cache all this stuff out to the model actually doing what the model spec says, you know, even on the exact
24:38example that uh on exa examples that they've literally published, you know, here's exactly what it should do and not do in this case, then I think we are just kind of a long way off from being ready to light a fuse in the spirit of Fourth of July fireworks that we, you know, we really don't know where this thing is is is pointed right now. Um, and so it makes me
25:05in some cosmic sense like excited, but in in a very like practical what do I expect to see over the next couple years sense like honestly pretty scared. Um, and I think that feeling is also growing. I guess we don't see it on display too much from Rune. I think Rune sort of has this like somewhat uh like biblical fear. It seems like he sort of um but he kind of doesn't let it bother
25:30him too much. I think others at the frontier companies are a little more emotionally consistent with these statements and I think that's what is driving a lot of these discussions about maybe we need a pause. You know, if we could coordinate a pause, maybe we would do it. Uh it probably would be good if if we could do it. So, yeah, it's it's getting real real. And I can't emphasize
25:56enough that I'm saying that not because I'm close-minded to change or very very different futures, but just because I don't think we have even the basic questions answered well at this point to give ourselves confidence that whatever we're about to create, if we hand off the AI R&D to the models themselves and uh let that process take us where it may, I don't I don't think we're particularly close to having
Related moments
Claim◆Article · 52 words
Banning open models now while closed models race 10-100X ahead in the hands of one or two companies creates a bigger long-term risk than allowing open Mythos-class models to exist.
Nathan Lambert · Interconnects
Claim◆Article · 50 words
Artificial superintelligence may arrive within a few years, and it is unclear whether alignment research is on track to be ready in that same timeframe, since current empirical programs at AI labs are unlikely to deliver a priori confidence that training an ASI will go well.
Sequent · Import AI