Coding agents given open-ended tasks like PyTorch make subtle but serious data-science mistakes such as information leakage, but with a proper high-level API (like Kumo's) they can accomplish the same task correctly in far fewer steps, showing agents need agent-friendly APIs to be effective.
Jure describes an internal test where an agent asked to build a fraud model from scratch made a subtle information-leakage bug (aggregating data 'till midnight' instead of the current time), but the same task took only ~50 correct lines when done through Kumo's higher-level API, illustrating that agents need agent-friendly abstractions to avoid multi-step failures. ✦ AI generated
Jure Leskovec · The TWIML AI Podcast · 2026-05-21 · original ↗
starts at this moment · 61:34
“So what's next?”
These models write thousands of lines of code but there are these super subtle data-sciency mistakes. So for example, we've done this together with Expedia, and the mistake the agent made was that when it created features for that given account it aggregated the transactions till midnight, not till the current time... that's information leakage. But if you give it a higher-level Kumo-like API then it's able to do the same work in about 50 lines of code, no mistakes.
verbatim transcript · starts at 61:34
61:34surfacing these two agents as uh as tools. Uh the second thing is right now right like the the coding agents are are uh out there but what we see is that coding agents require a proper abstraction and a proper infrastructure to be able to be effective right and and for example if you you could say hey why don't I just you know give this modeling task to clot code and clot code will
62:00build the model for me so you know what's the big deal and when we do that what we see we've run this internally is that these models write thousands of lines of code but there are this like super subtle data sciency mistakes. So for example, we we've done this uh together uh together uh with Expedia um and uh you know when when it was a account level fraud and uh mistakes for
62:31example the agent make was that when it created features for that given account it created it aggregated the transactions till midnight not till the current time right so it said oh today is I don't know uh uh April 30th. So we'll use the data up to midnight of April 30th. Not actually saying, "Hey, it's actually 10 10:00 a.m. on April 30th. We can only use data up to here." Right? So that's
62:57information leakage was a little mistake in there. Another mistake it made was that you know we did it at the transaction level instead of the account level. And these are like these subtle mistakes that really you need the human uh the human in there. But if you give it more this more like higher level kumo like API then it's able to do the same work in about 50 lines of code. No
63:22mistakes. >> The task in this case is to is to do what? Like I thought the task that you were describing was to code up something like what Kumo was trying to do. The task is build me a account level fraud uh fraud detection model over this data. >> And so what you're what you're proposing is like as opposed to trying to the agent trying to code it up from scratch,