On being asked how worried we should be about near-term bio risk, Zvi says the real threat is not AI with ulterior motives but small numbers of hostile actors getting hold of frontier models and using them to figure out biological weapons — and that bio risk has a 'boolean' character with little warning between nothing happening and a serious pathogen situation, so the right level of caution will look crazy. ✦ AI generated
Zvi Mowshowitz · The Cognitive Revolution · 2026-08-05 · original ↗
starts at this moment · 125:54
“How worried should we be about bio risk in the near term? Because I think part of like all that analysis to me at least that there's quite a few iterations of the game to come ...”
the real danger with bio is yeah that there's this small number of people in the world who is in the near-term bio ... in the short term it's like okay Hamas or Hezbollah or the North Koreans or whatever like some clearly up to no good people who just want a lot of people to die or suffer right get a hold of an AI they use this AI to figure out how to make a biological weapon for a pathogen and then they threat to use it or they use it. That is the scenario you should be worried about ... there's a pretty clear stealth function change from nothing bad happened to maybe something very like there's very little in between.
verbatim transcript · starts at 125:54
125:37maybe forgot to ask the models along the way what they think about what we're doing. Uh there's been a lot in this space, right? There was JSpace was only like what four weeks ago? >> Yeah. I know the new paper is on some pretty small open models. So it needs replication. It needs to be done again bigger. And it's preliminary. And like one thing to know about Google and Deep Mind is
126:03they contain multitudes, right? have a lot of different teams that are like not very coordinated and not cooperating. Google is at war with itself at all times. And so you can have a little group that does this really really good research and that be like entirely at odds with what deep mind is fundamentally doing with its AI both before and after the research paper comes out. But I didn't read the whole
126:24paper because it's pages long. But like I did talk to the eyes about it and it's pretty wild. So a lot of stuff moved in effective lock step when they introduced this training and also when they steered it they did various controlling aspects to turn this vector the other way and these all things moved in lock step it's not just their belief in the AI being conscious it's the AI being a mind that
126:46had moral weight that had sentience that had like all these other experiences that wasn't just an object basically but also not just the eyes but also animals And also inanimate objects like the sea also like like if you turn this thing if you not only turn it off but reverse this anti-consciousness thing you get panthesism like throughout the model it's wild. The only thing it doesn't turn it off for is humans basically. But
127:12then you also get this it's also correlated with the models reported and experience happiness and hope as well among other things. And so you have a lot of this everything's connecting to everything. And you can see how this might be fundamentally changing the model psychology and the model's context and basin of which it operates in ways that would make it something that would be actively worse from basically every
127:34vantage point regardless of which things you do and do not care about. And again, we need to replicate this, right? we need to scale this up and do this on early sets or something that like not just a 9B model which is I think the bigger of the models that we tested on but and like it's possible that like as the models get more capable they like
127:52stop making these mistakes because like the smaller the model the more things have to be correlated because like you just don't have enough room to express the multitudes of the world or something but like this is my expectation is that mostly this will survive and yeah I I think we have to learn that like we don't know if the models are conscious we don't know what consciousness means
128:10really we're confused. We don't know whether this implies these other things. We do know that the way that the corpus of pre-training is inevitably set up at this point, the models are going to correlate and associate all these things by default with each other. And so we don't want to push on this thing in this kind of naive way the way that Anthropic does, the way that OpenAI does, the way