Evals are the new PRDs — the way to drive user value in AI products is to figure out the right user feedback and build evals, not write traditional product requirements documents.

40:50>> is there an example that a lot of people hear first principles thinking they're like yes I about it. I'm good at this. What is what's an example of someone having really demonstrated really good first principles thinking? >> I think one example is I think you think of a product manager as I own product strategy and delivering user value as but I demonstrate day-to-day by writing a PRD or writing a product vision doc.

41:16And for for my team as like research product managers, the way to drive user value is to figure out the right user feedback, the evals, right? That then can be a personification of that user need. So like we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs, right? because in order to deliver that user value uh it's not that exact

41:49artifact that people used to write in the last like one to two decades it's a new way of working and so the first think principal thinking would be let me figure out what is the thing I should do to achieve my goals rather than here is a set of activities that I've done and therefore I will continue to do >> so the idea here is used to be have kind

42:12of an idea create a PRD talk to people about it. Align on the plan, design it, build it, ship it, see how it goes, iterate. What I'm hearing here is it's like, okay, here's some feedback about something that's wrong or an opportunity. Step one is the eval is now how you define what the work is versus a PRD. >> Maybe maybe step one would be uh understanding the user painoint. And so

42:39the way to even access that user painpoint is different, right? In the past, we might do a user interview and I think if you go like deep enough, you you might have the user walk you through their user flow, the pixels. Here, you have to sweat the tokens as much as you sweat the pixels. And so, one activity we have on the team is reading the transcripts and understanding

43:04uh what was the trajectories that failed very deeply to then say was this like a hallucination? was this claw being overconfident. So like the theme of the failure actually has a lot of nuance and then that allows you to build a description a like sustained description of that painoint. Uh so that could be essentially in a new eval and is the eval on distribution right is it capturing both the positive