Video · 2026-07-26 · 1h 34m · 8 moments

Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future

✦ AI generated

timeline · colored by role

Anecdote

In 2023 when I started, nobody said Anthropic and Claude and coding in the same sentence. I saw people were starting to use these models not just for code autocomplete, but actually writing long form code, and that was an opportunity for us to train Opus 3 to be better at coding. It ended up being a relatively smaller change from a training perspective but helped us differentiate competitively.

Diane identifies that early on, nobody associated Anthropic with coding, but she spotted users writing long-form code with models and pushed for Opus 3 to be trained on coding — a small training change that became a key differentiator.

transcript

Diane Penn: In 2023 when I started, nobody said anthropic and claude and coding in the same sentence. I think competitor models like GPT4 at the time was used a bit for coding but it was one of many use cases. And one thing that for example I saw was people are starting to use code these models not just for code autocomplete but actually writing long form code and is that an opportunity for us to train Opus 3 to be better at and it ended up being a relatively smaller change from a training perspective but it ended up helping us differentiate in the early days competitively for users and actually bring a lot of the very early Claude enthusiasts and developers because we were providing a value that they didn't really think was possible at the time.

gives example · 2

Claim◆

Opus 45 wouldn't have had that moment without a product like Claude Code, and Claude Code wouldn't have had that type of adoption accelerated without Opus 45. You need frontier products in order to have frontier models and for people to feel the magic of frontier models.

Diane describes the symbiotic relationship between Claude Code and Opus 45: the model reached an intelligence level where users could run things end-to-end in an agentic manner, but that magic required a great product vehicle to be felt, and the product needed the model to accelerate adoption.

transcript

Diane Penn: I think what was magical about Opus 45 is we also now not just had a model but a vehicle which is like a great product experience like Claude Code. One thing we say a lot on the team is you need frontier products in order to have frontier models and for people to feel the magic of frontier models. I think Opus 45 wouldn't have had that moment without a product like Claude Code and Claude Code I think wouldn't have had that type of adoption accelerated without Opus 45.

explains mechanism · 1extends · 1

Mechanism◆

There are discontinuous emerging capability jumps as you scale models — the models go from being unable to calculate something to reliably calculating it. These jumps are not perfectly predictable; you need the evals and systems to test for them, and that unpredictability is also what makes safety harder.

Diane explains that scaling laws produce smooth loss curves but discontinuous jumps in emerging capabilities — models unpredictably gain new abilities. This unpredictability is core to how the technology works and makes safety testing essential because you might not know a new capability exists until you test for it.

transcript

Diane Penn: There's some really interesting graphs in the original scaling law papers. Folks are very familiar with the scaling loss in the lens of as you add in more compute and data, the loss from next token prediction goes down. So it's a very smooth linear curve of the models get more intelligent as you scale them up. What's actually also interesting in that paper is there are these very different emerging capability graphs. For example, as you add in more data and you train the models with more compute, you see these actually discontinuous emerging capabilities jump. So the models go from 1 + 1 being a thing that it can't calculate to a thing that it can reliably calculate. And so these emerging capabilities, this nature of predictability is not necessarily everyone knows the exact moment — you need the evals to be able to assess that — has actually always been a part of how this technology works and also what makes things like safety harder because unless you have the eval, unless you have the systems to test, these jumps might actually happen and you don't know.

explains mechanism · 4extends · 2supports · 1

Prediction

If you're willing to spend $100,000 a year right now in tokens, you are living the way somebody in 2028 is going to live. You have to sweat the tokens as much as you sweat the pixels. You have to be using the models to come up with good and great and better ideas, and there's no substitute for that.

Diane endorses the idea that token-maxing today is a preview of the future, and emphasizes that the most creative thinkers and best prototypers internally spend a lot of time with Claude. She adds that experimentation doesn't have to be an individual sport — communal discovery in public channels accelerates finding new use cases.

transcript

Diane Penn: Internally some of the most creative thinkers, the best like prototypers do spend a lot of time with Claude with every new version of a research model that we have. And so there is something around you have to be like using the models to then come up with good then great then better ideas and there's no substitute for that. It's very hard to come up with a perfect strategy without touching the technology when it's moving this quickly.

extends · 1provides context · 1

Claim◆

You need frontier products in order to have frontier models, and for people to feel the magic of frontier models.

Diane explains the symbiotic relationship between frontier models and frontier products, using Claude Code and Opus 4.5 as an example.

transcript

Diane Penn: What was magical about Opus 45 is we also now not just had a model but a vehicle which is like a great product experience like Claude Code. One thing we say a lot on the team is you need frontier products in order to have frontier models and for people to feel the magic of frontier models. Opus 45 wouldn't have had that moment without a product like Claude Code and Claude Code wouldn't have had that type of adoption accelerated without Opus 45.

extends · 1

Context

One thing I ask the team frequently is: let's say Claude 8 comes around — what changes in what users do, and what does that mean for how you're building today? Is it going to be forward compatible to that experience? Being ambitious is very broad, so grounding it in thinking about future model capabilities is a way to describe that.

Diane describes how she grounds the team's ambition by asking them to imagine what Claude 8 will enable users to do, and then work backwards to ensure what they're building today is forward compatible with that future. This turns abstract ambition into concrete product strategy.

transcript

Diane Penn: One thing I ask the team frequently or how I think about when we're building a product is: let's say Claude 8 comes around — what changes in what users do, and then what does that mean for how you're building today? Is it going to be forward compatible to that experience, right? So like grounding it — I think being ambitious is very broad and so trying to ground it in some ways of describing that.

gives example · 2

Claim

Evals are the new PRDs. For my team as research product managers, the way to drive user value is to figure out the right user feedback and the evals that can be a personification of that user need. You have to sweat the tokens as much as you sweat the pixels.

Diane explains that for her product team, the traditional PRD is being replaced by evals as the core artifact. Understanding user pain points now means reading transcripts of model failures, identifying the precise failure mode (e.g., tool use vs. search synthesis), and building an eval set that researchers can act on to improve the model.

transcript

Diane Penn: For my team as like research product managers, the way to drive user value is to figure out the right user feedback, the evals, right? That then can be a personification of that user need. So like we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs, right? Because in order to deliver that user value, it's not that exact artifact that people used to write in the last one to two decades — it's a new way of working. And so the first principles thinking would be: let me figure out what is the thing I should do to achieve my goals rather than here is a set of activities that I've done and therefore I will continue to do.

provides context · 1

Claim◆

Evals are the new PRDs — the way to drive user value in AI products is to figure out the right user feedback and build evals, not write traditional product requirements documents.

Diane explains that for research product managers at Anthropic, evals have replaced PRDs as the core artifact for defining and measuring user value.

transcript

Diane Penn: The way to drive user value is to figure out the right user feedback, the evals, right? That then can be a personification of that user need. So like we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs, right? Because in order to deliver that user value it's not that exact artifact that people used to write in the last like one to two decades it's a new way of working.

extends · 1

Highlight slides

The Symbiotic Flywheel✦ from: Opus 45 wouldn't have had that moment without a product like Claude Code, and Claude Code wouldn't have had that type of adoption accelerated without Opus 45. You need frontier products in order to have frontier models and for people to feel the magic of frontier models.Frontier Products Unlock Frontier Models✦ from: Opus 45 wouldn't have had that moment without a product like Claude Code, and Claude Code wouldn't have had that type of adoption accelerated without Opus 45. You need frontier products in order to have frontier models and for people to feel the magic of frontier models.Discontinuous Capability Jumps in Scaling✦ from: There are discontinuous emerging capability jumps as you scale models — the models go from being unable to calculate something to reliably calculating it. These jumps are not perfectly predictable; you need the evals and systems to test for them, and that unpredictability is also what makes safety harder.Implications for Safety Testing✦ from: There are discontinuous emerging capability jumps as you scale models — the models go from being unable to calculate something to reliably calculating it. These jumps are not perfectly predictable; you need the evals and systems to test for them, and that unpredictability is also what makes safety harder.Frontier Models Need Frontier Products✦ from: You need frontier products in order to have frontier models, and for people to feel the magic of frontier models.A Symbiotic Relationship✦ from: You need frontier products in order to have frontier models, and for people to feel the magic of frontier models.Evals are the new PRDs✦ from: Evals are the new PRDs — the way to drive user value in AI products is to figure out the right user feedback and build evals, not write traditional product requirements documents.A new way of working✦ from: Evals are the new PRDs — the way to drive user value in AI products is to figure out the right user feedback and build evals, not write traditional product requirements documents.

Related episodes

Article · Import AI

Import AI 466: The bitter lesson for robotics, AIs complete week-long programming tasks; and OpenAI's accidental AI hacker

→

Video · a16z Podcast

Kavak's Playbook for Rebuilding a Company Around AI

→

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→