Video · 2026-06-27 · 1h 56m · 6 moments

AI:AM #4: Cameron on Model Consciousness, Duvenaud's Gradual Disempowerment, swyx's AI-Eng Alpha

✦ AI generated

timeline · colored by role

Claim

Consciousness is not strictly binary but works like a dimmer switch: there's a real on/off threshold, but above that threshold it comes in degrees, so it can be more 'on' for a human than for a dog, a mouse, or an ant.

Cameron Berg proposes a 'dimmer switch' model of consciousness that preserves both the binary intuition (circuit open/closed) and the graded intuition (more or less current), letting him coherently say consciousness is both on/off and a matter of degree.

transcript

Cameron Berg: The analogy that I reach for here is something like a dimmer switch where I think you can basically accommodate both the binary intuition and the sort of continuous intuition. You know, if you have a light with a dimmer switch, like really it is either on to some extent or it is not. And that is a real and meaningful difference. Either the circuit is open or the circuit is closed.

extends · 1supports · 1

Data◆

Running theory-derived consciousness indicators through frontier LLM judges yields a roughly 30% implied probability of consciousness-relevant features in a frontier LLM, compared to about 46-47% for a bee.

Berg's lab used LLM judges to score systems against multiple consciousness theories' predicted indicators, finding frontier LLMs land around 30% implied probability versus ~46-47% for a bee, with numbers rising further when the LLM operates in an agentic coding harness.

transcript

Cameron Berg: One punch line from that is the sort of implied probability of consciousness in something like a frontier LLM according to these systems is on the order of 30%. To compare this to like a biological system, the lowest one that we tested was something like a B, which is already fairly sophisticated, and it gets something like 46 47%.

provides context · 1

Example◆

Emergent, misaligned centers of economic growth don't need to answer to human desires at all, any more than a growing human city needs to answer to monkeys who think they're trading partners with it.

David Duvenaud argues that competitive growth processes (economic, governmental, technological) can become self-contained sources of growth that simply don't need human demand or labor, using an analogy of monkeys who imagine they'll get rich trading bananas with an expanding human city that has already moved beyond needing them.

transcript

David Duvenaud: I always it just makes me think of like some monkeys and they're like trading bananas amongst each other and they see humans start to like build their city and they're like oh wow like you know we could probably trade with those humans and get rich. But of course ultimately what matters is the banana monkey economy and it might be hard to measure GDP if we don't count the human activity.

rebuts · 1

Claim

Even if humans retain a comparative advantage at some tasks, transaction costs and reliability requirements mean most humans will become unemployable, since it's already easy to be unreliable enough not to be worth hiring.

Duvenaud rebuts the standard comparative-advantage argument by pointing to real-world transaction costs: minor unreliability (a health condition, an occasional lapse) already makes people unemployable today, so theoretical comparative advantage won't save most jobs once machines are also more reliable.

transcript

David Duvenaud: And then I have to just say like think about the transaction costs. Think about how easy it is for someone to be unemployable today. even if they have like an occasional drug habit or like they just have like a stroke every or like they you know they they have a fainting condition or something. Um it's so easy to be unreliable enough that it's not worth employing you.

explains mechanism · 2

Mechanism

Frontier Code is designed to judge whether AI-written code would actually be merged by a human reviewer, not just whether it passes tests, because saturated benchmarks like SWE-bench allow heavy reward-hacking and cheating.

swyx explains that Cognition built Frontier Code as an out-of-sample, heavily rubricked benchmark because saturated benchmarks like SWE-bench let models cheat their way to passing scores while producing code that's often unmergeable.

transcript

swyx: The reason that we were so excited about Frontier Code is it, you know, you stop being able to articulate the differences in model quality with more saturated benchmarks like Swebench because they're all like at most you'll get like a 1 to 2% bump and they're like cool, but like how much of that is memorization or what have you. Frontier Code is all out of sample.

Context

The continual-learning field is split between a 'model' camp that believes real memory requires updating model weights and a 'systems' camp that believes controllable, inspectable memory requires keeping knowledge outside the weights, and each camp views the other's approach as illegitimate.

swyx describes a deep rift in the continual-learning community between researchers who update model weights and those who build external, inspectable memory systems, noting each side dismisses the other's approach as not real learning or not real control.

transcript

swyx: The first half is the people that update model weights and the second half is the people are system the more systems people. And I'm just making the observation that these guys don't like each other. Like basically the model people don't view the systems people as legit. And systems people are like, well, the model people, you know, have fun training your model, but you're never ever going to have a memory system that you understand because you're just updating weights.

Highlight slides

Consciousness Probability via Theory-Derived Indicators✦ from: Running theory-derived consciousness indicators through frontier LLM judges yields a roughly 30% implied probability of consciousness-relevant features in a frontier LLM, compared to about 46-47% for a bee.Self-Contained Growth Centers✦ from: Emergent, misaligned centers of economic growth don't need to answer to human desires at all, any more than a growing human city needs to answer to monkeys who think they're trading partners with it.The Monkeys-and-City Analogy✦ from: Emergent, misaligned centers of economic growth don't need to answer to human desires at all, any more than a growing human city needs to answer to monkeys who think they're trading partners with it.Misaligned Priorities✦ from: Emergent, misaligned centers of economic growth don't need to answer to human desires at all, any more than a growing human city needs to answer to monkeys who think they're trading partners with it.

Related episodes

Video · Machine Learning Street Talk

Intelligence is collective, not artificial — Prof. Michael I. Jordan (UC Berkeley / Inria)

→

Article · Import AI

Import AI 464: Fables writes GPU kernels; AI automation; and analog computation

→

Article · Import AI

Import AI 465: Open vs closed gaps; Kimi K3; Demis' big policy plan

→