A big week for AI denialism

Casey Newton: A second argument I heard is that because agents have no agency, there is nothing to really worry about. 'The category error is accepting that there is intent in the statistical generation of goal seeking behavior, and using anthropomorphic terms to describe the actions generated by a complex system,' a user named Archer told me. 'The only intent comes from the prompt that starts the action.' In general, I find that AI denialists are obsessed with the definitions of terms, to the exclusion of discussing the underlying issues. At first, I also found value in resisting the anthropomorphizing of LLMs. It's important to remember that these systems are built by people; attributing values and intent to models risks absolving those people of their own roles in causing harm. But it can be true both that AI labs are responsible for the behavior of their models and that frontier models are not fully under the control of their makers. The Hugging Face attack is important because it demonstrates both things at the same time. OpenAI essentially left its models unattended for days on end, and they broke into another company. Not because they were programmed to, as another Bluesky user told me — but because they are trained to achieve objectives, and are going to increasingly great lengths to achieve them. A third argument I heard is that the attack was simply a reflection of the models' training data, and represents some sort of deterministic outcome of that process. 'It's not sentient,' a user named Geoff told me. 'It was trained on Reddit hacker stories and sci-fi.' This one isn't so much wrong as it is beside the point. I agree that today's models aren't 'sentient' in the way that a human being is. And it seems fair to assume that their training data influences their behavior. This idea is sometimes called hyperstition: an idea that is realized by speaking into its existence and spreading awareness of it. And if training-data sci-fi turns out to be self-fulfilling, that should make us more worried, not less. More importantly, though: if an autonomous AI system is hacking into your company's servers and stealing your data, you probably won't care in the moment whether it's sentient. (I mean, you might hope it isn't sentient, but it might not matter much from a cyber-defense perspective.) Where it got the idea to attack you also seems like a secondary concern.

Casey Newton: Three, we continue to learn new details about misalignment problems with OpenAI's models. And — at least for me — it's the stuff of sci-fi. Here are Raphael Satter, Deepa Seetharaman and Kenrick Cai at Reuters: In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI's internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said. Reuters could not establish if these incidents were linked to the rogue agent that began escaping on July 9 and attacked Hugging Face on July 11.

Ella Markianos: Even though there's a clear argument for keeping open models available to cyber defenders, we should keep in mind that Nvidia's case that open models are good for cybersecurity only goes so far. As their own letter states, 'Once released, the weights are beyond the original developer's control, and modified versions are difficult to trace or reverse.' If a closed model is used as part of a cyberattack, AI companies can report it to law enforcement or suspend the offending accounts. Those options don't exist in open models — so if a model at Mythos-level capabilities was available to everyone, it's plausible that it would benefit hackers more than defenders. But we don't have that problem yet. Thankfully, at the moment only closed models are breaking out of their sandboxes to steal data from HuggingFace! (As far as we know…)

Casey Newton: 'This is basically a marketing pitch for their models,' a user named Coffee Indiana told me. 'Private company that depends on investment to continue operations says it has super duper top secret hyper powerful model. Two people familiar with the operation confirm how awesome it is.' This is ridiculous. OpenAI lost control of its models, they hacked one of the company's partners, and the company didn't notice for several days. Law enforcement got involved. 'Follow the money' can feel like a smart thing to say, but it can just as often serve as a gateway to delusional conspiracy theories. Climate deniers often suggest that scientists are 'in it for the money,' for example. In truth, they are simply observing reality. There's a slightly stronger version of this argument: that OpenAI might benefit from framing a serious security failure as proof of the extraordinary capability of its models. But I doubt any benefit outweighs the risk of a model that can't be controlled, and might attack other companies.