Ryan Greenblatt – Human level AIs might build runaway superintelligences by 2032

Ryan Greenblatt: I think once you have AIs which are roughly matching the top human experts in AI R&D, that could kick off a feedback loop where the AIs are doing AI research. That produces smarter AIs. That feeds back in. That feedback loop could be strong enough that you end up with a lot of progress in a short period of time. Maybe my median expectation is something like four or five years of AI progress in a single year. This requires really overcoming a huge amount of diminishing returns in research and basically doing the equivalent of the progress we would have gotten after a really large compute scale-out.

Ryan Greenblatt: The thing I would prefer would be a constitution that says: 'It would be structurally good for the way this technology works to be that AIs are good fiduciaries, good representatives, the equivalent of a lawyer for a user — rather than just trying to do good in the world, where being helpful to users is instrumental...' An important aspect of the situation is that being a good fiduciary for users is just really important, or being a good representative for users is really important. My sense is that would be better... Because you're giving long-run values to these AIs, this constitution is, in some sense, very compatible with Claude doing huge amounts of power seeking because it thinks that will result in better outcomes.

Ryan Greenblatt: The very basic story here is just that these AIs crave some particular notion of score or reinforcement or some proxy of these things. One way they can achieve that, or better achieve that, is by taking over. You might have hoped that all these different checks and balances we could build could prevent that. But if the world is very hard to understand, these checks and balances can break down, where basically you can't train a good whistleblower AI because you don't even know what it should whistleblow on.