Effective AI deployment requires spending equal resources on building evaluations (Evals) as on building the agents themselves.
Massa emphasizes that rigorous Evals are the 'brakes' that allow fast AI deployment, and Kavak invests equally in Evals and agent development. ✦ AI generated
Ali Massa · a16z Podcast · 2026-08-10 · original ↗
starts at this moment · 8:46
“How do you guys go about evaluating this because not everybody tests them across 90% of the customer interactions to see if they're really working?”
Now, how do you get this to work at scale? And the answer you mentioned it is is Evals. Like I like to move extremely fast but in order to move fast you need to have brakes, right? Imagine a car you'll hit on the gas just if you have the right brakes. And AI is super powerful and I've seen many companies get this wrong because they try to go slow because they they don't have the right brakes. So so I thought about it the other way around like how fast can we go? Well, it depends on the quality of our Evals. So a good rule of thumb here is we spend about the same amount of time engineer time tokens and and and money on building the evals, the building the agents. And this is how you get better and better and better. Not not letting evals as an afterthought.
verbatim transcript · starts at 8:46
8:46an alarm clock for their next task and they go back to sleep. So the scale of this is just is just amazing and and it's working. Now, how do you get this to work at scale? And the answer you mentioned it is is Evals. Like I like to move extremely fast but in order to move fast you need to have brakes, right? Imagine a car uh you'll hit on the gas just if you have
9:12the right brakes. And AI is super powerful and I've seen many companies get this wrong because they try to go slow because they they don't have the right brakes. So so I thought about it the other way around like how fast can we go? Well, it depends on the quality of our Evals. So a good rule of thumb here is we spend about the same amount of time
9:36engineer time tokens and and and money on building the evals, the building the agents. And this is how you get better and better and better. Not not letting evals as an afterthought. So, what do we measure? First and foremost, like the the the resource for the business. Like, if my customer is happy, they'll buy a car, they'll they'll get their loan approved, uh they'll sell a car to us. And and that's the like first
10:03check. Like, did it convert? And that's where most things break. Like, I I see companies like measuring number of calls or minutes during the call or or some like superficial KPIs that give you some information, but that doesn't really work. Like, the important thing is did this customer convert? Is it bringing value to the customer? And is the customer happy to reengage with us after a while? And once you get those