Evals are as important as the agents themselves—Kavak spends equal engineering time on evaluation systems as on building the agents, focusing on conversion metrics rather than superficial KPIs.
Massa emphasizes that evals act as brakes allowing faster deployment, with the company spending equal resources on evaluation and agent development, measuring actual customer conversion. ✦ AI generated
Ali Massa · a16z Podcast · 2026-08-10 · original ↗
starts at this moment · 9:36
“One of the questions is like how do you how do you guys go about evaluating this because not everybody tests them across 90% of the customer interactions to see if they're really working.”
A good rule of thumb here is we spend about the same amount of time engineer time tokens and and and money on building the evals, the building the agents. And this is how you get better and better and better. Not not letting evals as an afterthought. So, what do we measure? First and foremost, like the the the resource for the business. Like, if my customer is happy, they'll buy a car, they'll they'll get their loan approved, uh they'll sell a car to us. And and that's the like first check. Like, did it convert? And that's where most things break. Like, I I see companies like measuring number of calls or minutes during the call or or some like superficial KPIs that give you some information, but that doesn't really work. Like, the important thing is did this customer convert? Is it bringing value to the customer? And is the customer happy to reengage with us after a while?
verbatim transcript · starts at 9:36
9:36engineer time tokens and and and money on building the evals, the building the agents. And this is how you get better and better and better. Not not letting evals as an afterthought. So, what do we measure? First and foremost, like the the the resource for the business. Like, if my customer is happy, they'll buy a car, they'll they'll get their loan approved, uh they'll sell a car to us. And and that's the like first
10:03check. Like, did it convert? And that's where most things break. Like, I I see companies like measuring number of calls or minutes during the call or or some like superficial KPIs that give you some information, but that doesn't really work. Like, the important thing is did this customer convert? Is it bringing value to the customer? And is the customer happy to reengage with us after a while? And once you get those
10:31evals connected, then it's just optimizing the right agent and the tech architecture and giving the agent skills to to scale this and and cater to millions of customers. >> It's really really amazing. And you know, related to this is like, okay, so you create the right evals, you know it's working. You know, some people some companies still feel a little bit risk averse in putting them in front of of of the
10:55customers and being able to perform >> Mhm. >> the highest leverage task, which in your case would be selling. Do your agents really sell to customers? >> Yes. So, so we never built customer support or customer service agents. We we we built like sales agents. It's extremely hard to sell a car in in Latin America. So, imagine someone wanting to buy a car, they can choose like among like 20,000
11:21SKUs. Then they need to pick like financing and go through the financing process, insurance, and and coverage. And then they're probably trading in their car. So, so we need to quote that car. So, it's a process that if someone does it or or the way Kavak did it back in in in 2020 2021, was you need to be extremely good at 15 different things and have 15