Product management in the AI era has shifted: evals are the new PRDs, and PMs must 'sweat the tokens' as much as they sweat the pixels.
Dianne describes how her team's way of driving user value has shifted from writing PRDs to building evals — creating testable, reproducible representations of user pain points that researchers can act on, because the key failures now live in token trajectories, not UI flows. ✦ AI generated
Dianne Penn · Lenny's Podcast · 2026-07-26 · original ↗
starts at this moment · 39:55
“What are some things that you notice? What are you looking for more most? What's kind like trending up in what you find is important and what's kind of trending down?”
The way to drive user value is to figure out the right user feedback, the evals, right? That then can be a personification of that user need. So like we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs... Here, you have to sweat the tokens as much as you sweat the pixels. And so, one activity we have on the team is reading the transcripts and understanding what was the trajectories that failed very deeply to then say was this like a hallucination? was this claw being overconfident.
verbatim transcript · starts at 39:55
39:55today's world, what are some things that you notice? What are you looking for more most? What are you looking for more? What's kind like trending up in what you find is important and what's kind of trending down? We actually on my team have not changed our hiring loop uh for three years now. Um so what we actually look for and the traits and how we evaluate uh
40:19generalists like PM's generalist like research product managers have actually been the same. Um so I think some of those traits number one is first principles thinking and this is really uh rather than pattern matching what you used to do in let's say consumer product or B2B SAS um but actually figuring out in this moment for this user group with this technology what what is the user value
40:50>> is there an example that a lot of people hear first principles thinking they're like yes I about it. I'm good at this. What is what's an example of someone having really demonstrated really good first principles thinking? >> I think one example is I think you think of a product manager as I own product strategy and delivering user value as but I demonstrate day-to-day by writing a PRD or writing a product vision doc.
41:16And for for my team as like research product managers, the way to drive user value is to figure out the right user feedback, the evals, right? That then can be a personification of that user need. So like we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs, right? because in order to deliver that user value uh it's not that exact
41:49artifact that people used to write in the last like one to two decades it's a new way of working and so the first think principal thinking would be let me figure out what is the thing I should do to achieve my goals rather than here is a set of activities that I've done and therefore I will continue to do >> so the idea here is used to be have kind
42:12of an idea create a PRD talk to people about it. Align on the plan, design it, build it, ship it, see how it goes, iterate. What I'm hearing here is it's like, okay, here's some feedback about something that's wrong or an opportunity. Step one is the eval is now how you define what the work is versus a PRD. >> Maybe maybe step one would be uh understanding the user painoint. And so
42:39the way to even access that user painpoint is different, right? In the past, we might do a user interview and I think if you go like deep enough, you you might have the user walk you through their user flow, the pixels. Here, you have to sweat the tokens as much as you sweat the pixels. And so, one activity we have on the team is reading the transcripts and understanding
43:04uh what was the trajectories that failed very deeply to then say was this like a hallucination? was this claw being overconfident. So like the theme of the failure actually has a lot of nuance and then that allows you to build a description a like sustained description of that painoint. Uh so that could be essentially in a new eval and is the eval on distribution right is it capturing both the positive