Evals should receive equal engineering investment as the agents themselves, and the only evals that matter are whether customers actually convert and come back—superficial metrics like call duration or number of calls are misleading.
Ali emphasizes that evals are the brakes that let you go fast—Kavak spends roughly equal time and resources building evals as building agents, and measures conversion and customer re-engagement rather than superficial KPIs. ✦ AI generated
Ali Massa · a16z Podcast · 2026-08-10 · original ↗
starts at this moment · 9:36
“One of the questions is like how do you how do you guys go about evaluating this because not everybody tests them across 90% of the customer interactions to see if they're really working”
A good rule of thumb here is we spend about the same amount of time engineer time tokens and and and money on building the evals, building the agents. And this is how you get better and better and better. Not not letting evals as an afterthought. So, what do we measure? First and foremost, like the the the metric for the business. Like, if my customer is happy, they'll buy a car, they'll they'll get their loan approved, uh they'll sell a car to us. And and that's the like first check. Like, did it convert? And that's where most things break. Like, I I see companies like measuring number of calls or minutes during the call or or some like superficial KPIs that give you some information, but that doesn't really work. Like, the important thing is did this customer convert? Is it bringing value to the customer? And is the customer happy to reengage with us after a while?
verbatim transcript · starts at 9:36
9:36engineer time tokens and and and money on building the evals, the building the agents. And this is how you get better and better and better. Not not letting evals as an afterthought. So, what do we measure? First and foremost, like the the the resource for the business. Like, if my customer is happy, they'll buy a car, they'll they'll get their loan approved, uh they'll sell a car to us. And and that's the like first
10:03check. Like, did it convert? And that's where most things break. Like, I I see companies like measuring number of calls or minutes during the call or or some like superficial KPIs that give you some information, but that doesn't really work. Like, the important thing is did this customer convert? Is it bringing value to the customer? And is the customer happy to reengage with us after a while? And once you get those
10:31evals connected, then it's just optimizing the right agent and the tech architecture and giving the agent skills to to scale this and and cater to millions of customers. >> It's really really amazing. And you know, related to this is like, okay, so you create the right evals, you know it's working. You know, some people some companies still feel a little bit risk averse in putting them in front of of of the
10:55customers and being able to perform >> Mhm. >> the highest leverage task, which in your case would be selling. Do your agents really sell to customers? >> Yes. So, so we never built customer support or customer service agents. We we we built like sales agents. It's extremely hard to sell a car in in Latin America. So, imagine someone wanting to buy a car, they can choose like among like 20,000