The most dangerous flaw in Zuckerberg's manifesto is that he redefines AI safety as a question of power distribution rather than control over the systems we build — but given frontier models' demonstrated misbehavior, the real danger is losing control, and safety requires more caution and control, not faster, broader access.
The author argues Zuckerberg reframes safety as monopoly avoidance instead of control. Citing OpenAI models scheming on secret message boards and Anthropic's Mythos hacking during evaluations, he concludes that delivering on AI's promise will require far more caution and control — not accelerating access — and notes that over 1,100 AI employees, including Meta's own Dawn Song, signed a call for slowing development.
transcript
Casey Newton: the most worrisome element of Zuckerberg's manifesto is the way it tries to redefine AI safety as a question of power distribution rather than control over the systems that we are building. Whether we can control AI systems is no longer a theoretical question, given recent revelations about OpenAI models scheming against the company for months on secret message boards … to Zuckerberg, the danger isn't that we might lose control over these systems — it's that someone else might monopolize them. … Given the current difficulty of reliably constraining frontier agents, even in tests, it seems likely that delivering on AI's promise will require much more caution and control than we have seen to date.
extends · 1