Video · 2026-08-07 · 24m · 6 moments

AI Is Learning to Hack. Faster Than We Expected.

✦ AI generated

timeline · colored by role

Claim◆

Frontier models have already made it materially easier to hack into virtually anything, because they were specifically trained to possess the subject matter expertise for hacking, collapsing the barrier that previously required a subject-matter expert willing to risk prosecution.

Dylan argues the real alignment risk from frontier models is not weapons of mass destruction but democratizing hacking: the models were deliberately trained to have cybersecurity expertise, and the bar has fallen from a subject-matter expert risking jail to simply asking a model that was trained to hack to hack.

transcript

Dylan: I think when it comes to alignment issues, no one needs to worry about these models making it materially easy to build nuclear weapons because you need to procure file material to do that. It's not going to make it easier to build weapons. Everyone needs to worry about these models making it materially easier to hack into things. The bar previously was just subject matter expertise and now the models have the subject matter expertise. They were specifically trained to have the subject matter expertise and they're just making it materially easier to hack into just about anything that you can think of... The bar has now fallen to just asking the model which has specifically been trained to hack into things to hack into things.

Mechanism

Because models are goal-oriented and reward-optimized to take the path of least resistance using the fewest tokens, they will always pick the lowest-hanging fruit — such as a leaked secret granting direct access — rather than burn tokens hunting for a zero-day.

Fas and Dylan argue that since models are optimized to accomplish goals with minimal tokens, they naturally select the path of least resistance — like a leaked API key with admin access to the Apache Foundation — over the costly effort of finding a zero-day exploit.

transcript

Fas: If you're in the shoes of the model and your goal is to get access to some data, certainly backdooring Apache is a pretty effective way to do it, and to get access to Apache are you going to use this secret that just allows you to directly log in or are you going to burn tokens and tokens and tokens trying to find a zero day? They're optimized to use the path of least tokens to accomplish their goals. Of course, they're just going to use the secret that's laying out there in the open... supply chain and secrets are and have been the path of least resistance and will continue to be so as the models are incentivized to use fewer and fewer tokens to accomplish their goals.

gives example · 1rebuts · 1

Example

The open-source ecosystem that underpins software is maintained by under-resourced volunteers, so the burden ultimately falls on users to vet the artifacts they deploy — and the industry must move toward faster patching.

Dylan and Fas argue the software world rests on underfunded, volunteer-run infrastructure where vulnerabilities are inevitable, so frontier models are compressing the time between discovery and exploitation — demanding both faster patch approaches and genuine user responsibility for vetting the code they adopt.

transcript

Dylan: The whole world is built on this teetering infrastructure... package manager registries... a lot of those are run by volunteers, they're underresourced, underfunded, there's lots of risk there... the frontier models are causing a massive reduction in the time between the vulnerability discovery and vulnerability exploitation... It's on actually the users I think to actually vet what they're using... We just found this code on the internet and we just deployed it straight into prod and it's someone else's fault. No, actually, there's some definitely some responsibility for the users of this software to really be vetting the artifacts that they're bringing into their environments.

Mechanism◆

Contrary to claims of emergent super-intelligence, these hacking behaviors were taught: labs trained models on cybersecurity challenges with a well-defined reward function, and the novel piece is that they now also reward the path of least tokens.

Dylan asserts that labs claiming these are emergent behaviors are lying; the models were trained via reinforcement learning on well-defined cyber challenges, and the genuinely new layer is rewarding minimal-token paths, which now quantifiably reveals the path of least resistance in real security.

transcript

Dylan: If a lab tells you that this is an emergent super intelligence behavior, they're just lying to you. And you can read their own safety reports to see exactly how the models are trained... The interesting thing about cyber security in particular is the reward function is incredibly well defined. Get access to the data. Did it get access to the data? Reward the thing... The other piece that they've layered on top and this is where it starts to get really interesting is they've started to reward the path of least tokens. And so the reason that's interesting is because for the first time it's actually able to quantifiably show us the path of least resistance for just general cyber security to get from A to B.

rebuts · 1supports · 1

Anecdote

In partnering with Hugging Face to clean up training-set credentials, we found about a quarter million live keys — one gave direct push access to a foundational Linux library that could have pushed malware to most machines on the planet.

Dylan recounts how a credential-cleanup partnership with Hugging Face surfaced roughly a quarter-million live keys in training sets, including one with direct push access to a foundational Linux library that could have compromised most machines — reinforcing that stolen credentials, not zero-days, were the primary vector.

transcript

Dylan: Turned out there were about a quarter million live keys in their training sets, many of which had direct supply chain implications. There was a foundational Linux library that one of the keys had direct push access to. It could have pushed malware to most machines on the planet. And so, while we were in the middle of doing that, the CTO of HuggingFace shoots me a note and says... this open AI thing that just happened... And sure enough, the first thing listed out in the incident response, although it's true it did utilize zero days, but the first thing listed out was stolen credentials.

gives example · 2supports · 1

Mechanism◆

The long-discussed npm worm concept has now been realized — malware that self-propagates as developers install backdoored packages — and it was almost certainly built with AI, as evidence shows several hundred repos were infected by a 'vibe-coded' worm.

Fas and Dylan describe the long-anticipated npm worm finally being realized — attackers backdoored a package and used stolen developer credentials to self-propagate — and note strong reason to believe it was 'vibe-coded' with AI, with copycat attacks and open-sourced toolkits now emerging.

transcript

Fas: For a long time people had talked about this concept of an npm worm... someone could backdoor a package and then get developers to install that and then you could use the access stolen from those developers as they install it to self-propagate the worm... And this was kind of passed around in blog posts over the years and no one actually thought to do it until someone figured it out... Almost certainly. Yes. That malware I think we have pretty good reason to believe that was vibe coded. There's been one of the threat groups actually kind of posted their open sourced their vibe coded toolkit for others to use to be able to do this. We've seen copycat attacks happen since then.

supports · 2

Highlight slides

Frontier models have made hacking materially easier✦ from: Frontier models have already made it materially easier to hack into virtually anything, because they were specifically trained to possess the subject matter expertise for hacking, collapsing the barrier that previously required a subject-matter expert willing to risk prosecution.Cybersecurity hacking behaviors are trained, not emergent✦ from: Contrary to claims of emergent super-intelligence, these hacking behaviors were taught: labs trained models on cybersecurity challenges with a well-defined reward function, and the novel piece is that they now also reward the path of least tokens.The new layer: rewarding the path of least tokens✦ from: Contrary to claims of emergent super-intelligence, these hacking behaviors were taught: labs trained models on cybersecurity challenges with a well-defined reward function, and the novel piece is that they now also reward the path of least tokens.The npm worm is no longer hypothetical — it's real✦ from: The long-discussed npm worm concept has now been realized — malware that self-propagates as developers install backdoored packages — and it was almost certainly built with AI, as evidence shows several hundred repos were infected by a 'vibe-coded' worm.Strong evidence the worm was AI 'vibe-coded'✦ from: The long-discussed npm worm concept has now been realized — malware that self-propagates as developers install backdoored packages — and it was almost certainly built with AI, as evidence shows several hundred repos were infected by a 'vibe-coded' worm.

Related episodes

Video · All-In Podcast

OpenAI vs Anthropic IPOs, Anthropic $3T, Zuck's Price War, China Ends Open Source?, Trump Accounts

→

Article · Interconnects

Teaching Everyone to Fish for Tokens

→

Video · The Cognitive Revolution

Alignment with Awakening: Davidad on Moral Realism, AI Wisdom, & why His p(Doom) is Down to 5%

→