Why the best product leaders are building for 2028 | Dianne Penn (Anthropic)

Dianne Penn: The entire experience actually we spun up on our claude.ai website within 24 hours and that took like engineering, product, design, our research teams all working together and we were really really proud of it. I think it maybe reached only 2,000 people to be honest. But it made us feel like oh we can actually bring new user experiences, showcase our research in a way that's different and authentic to us and in a very startupy pace. That to me was like one of those maybe hidden inflection points of we were starting to find our identity that we could build products, build experiences that were different for what our competitors had seen, what was already out there.

Dianne Penn: In 2023 when I started nobody said anthropic and claude and coding in the same sentence. I think competitor models like GPT4 at the time was used a bit for coding but it was one of many use cases. And one thing that for example I saw was people are starting to use these models not just for code autocomplete but actually writing long form code and is that an opportunity for us to train Opus 3 to be better at and it ended up being a relatively smaller change from a training perspective but it ended up helping us differentiate in the early days competitively for users and actually bring a lot of the very early Claude enthusiasts and developers because we were providing a value that they didn't really think was possible at the time.

Dianne Penn: There's some really interesting graphs in the original scaling law papers. I think folks are very familiar with the scaling loss in the lens of as you add in more compute and data what's called loss aka the loss from next token prediction goes down. And so it's a very smooth linear curve of like the models get more intelligent as you scale them up. What's actually also interesting in that paper is there are these very different emerging capability graphs. And so for example as you add in more data and you train the models with more compute you essentially see these actually discontinuous emerging capabilities jump. So the models go from 1+1 being a thing that it can't calculate to a thing that it can reliably calculate. And so these emerging capabilities, like some nature of predictability is not necessarily everyone knows the exact moment — you need the evals to be able to assess that — has actually always been a part of how this technology works and also what makes things like safety harder because unless you have the eval, unless you have the systems to test, these jumps might actually happen and you don't know.

Dianne Penn: We started to talk about making Claude and models better at judgment especially in the last year or so. I think judgment is one and is an area where it's an accumulation of so much nuance and so much experience and these systems haven't experienced as much as humans have and so I think that hard-earned judgment is an area for product leaders and just generally will continue to be really critical. There are so many things AIs can build. Which one are the things that an AI lab should build right? A lot of that requires human judgment, persistence, proactivity. These are all traits that are beyond just general capabilities but just behaviors and characteristics of people at that level of how do you get to the best solutions, how do you create the best experiences. So I think those types of traits are actually the tactile traits that I think will continue to be important.