Evals are the new PRDs. For my team as research product managers, the way to drive user value is to figure out the right user feedback and the evals that can be a personification of that user need. You have to sweat the tokens as much as you sweat the pixels.
Diane explains that for her product team, the traditional PRD is being replaced by evals as the core artifact. Understanding user pain points now means reading transcripts of model failures, identifying the precise failure mode (e.g., tool use vs. search synthesis), and building an eval set that researchers can act on to improve the model. ✦ AI generated
Diane Penn · Lenny's Podcast · 2026-07-26 · original ↗
starts at this moment · 39:53
“What are you looking for more most? What are you looking for more? What's kind like trending up in what you find is important and what's kind of trending down?”
For my team as like research product managers, the way to drive user value is to figure out the right user feedback, the evals, right? That then can be a personification of that user need. So like we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs, right? Because in order to deliver that user value, it's not that exact artifact that people used to write in the last one to two decades — it's a new way of working. And so the first principles thinking would be: let me figure out what is the thing I should do to achieve my goals rather than here is a set of activities that I've done and therefore I will continue to do.
verbatim transcript · starts at 39:53
39:55today's world, what are some things that you notice? What are you looking for more most? What are you looking for more? What's kind like trending up in what you find is important and what's kind of trending down? We actually on my team have not changed our hiring loop uh for three years now. Um so what we actually look for and the traits and how we evaluate uh
40:19generalists like PM's generalist like research product managers have actually been the same. Um so I think some of those traits number one is first principles thinking and this is really uh rather than pattern matching what you used to do in let's say consumer product or B2B SAS um but actually figuring out in this moment for this user group with this technology what what is the user value
40:50>> is there an example that a lot of people hear first principles thinking they're like yes I about it. I'm good at this. What is what's an example of someone having really demonstrated really good first principles thinking? >> I think one example is I think you think of a product manager as I own product strategy and delivering user value as but I demonstrate day-to-day by writing a PRD or writing a product vision doc.
41:16And for for my team as like research product managers, the way to drive user value is to figure out the right user feedback, the evals, right? That then can be a personification of that user need. So like we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs, right? because in order to deliver that user value uh it's not that exact
41:49artifact that people used to write in the last like one to two decades it's a new way of working and so the first think principal thinking would be let me figure out what is the thing I should do to achieve my goals rather than here is a set of activities that I've done and therefore I will continue to do >> so the idea here is used to be have kind