Jure recounts an internal test with Expedia where a coding agent building an account-level fraud model from scratch made a subtle information-leakage bug (aggregating transactions to midnight instead of the current time), arguing agents need agent-friendly, higher-level APIs to avoid such mistakes. ✦ AI generated
Jure Leskovec · The TWIML AI Podcast · 2026-05-21 · original ↗
starts at this moment · 62:00
“So what's next?”
When we do that what we see we've run this internally is that these models write thousands of lines of code but there are this like super subtle data sciency mistakes. So for example, we we've done this uh together uh together uh with Expedia um and uh you know when when it was a account level fraud and uh mistakes for example the agent make was that when it created features for that given account it created it aggregated the transactions till midnight not till the current time.
verbatim transcript · starts at 62:00
62:00build the model for me so you know what's the big deal and when we do that what we see we've run this internally is that these models write thousands of lines of code but there are this like super subtle data sciency mistakes. So for example, we we've done this uh together uh together uh with Expedia um and uh you know when when it was a account level fraud and uh mistakes for
62:31example the agent make was that when it created features for that given account it created it aggregated the transactions till midnight not till the current time right so it said oh today is I don't know uh uh April 30th. So we'll use the data up to midnight of April 30th. Not actually saying, "Hey, it's actually 10 10:00 a.m. on April 30th. We can only use data up to here." Right? So that's
62:57information leakage was a little mistake in there. Another mistake it made was that you know we did it at the transaction level instead of the account level. And these are like these subtle mistakes that really you need the human uh the human in there. But if you give it more this more like higher level kumo like API then it's able to do the same work in about 50 lines of code. No
63:22mistakes. >> The task in this case is to is to do what? Like I thought the task that you were describing was to code up something like what Kumo was trying to do. The task is build me a account level fraud uh fraud detection model over this data. >> And so what you're what you're proposing is like as opposed to trying to the agent trying to code it up from scratch,
63:50you create some kind of skill or something like that that teaches it how to use Kumo to get the same information. >> Yeah. Or what I'm saying is agents you know they they can go they can autonomously maybe make two steps but not 100 steps. So now when I ask it for a task I can say hey here's PyTorch go build me the model that that's you know takes thousand