Analytics discovers patterns by finding traces that are mathematically different from the rest of the distribution, then using LLMs to explain whether the difference is good or bad.

13:15a very unsupervised learning approach. It turns out it's very difficult without any prior information to say, "Is A better than B?" And being able to say like this was a bad agent coup or something like that. But mathematically, it's actually relatively straightforward to ask the question, is A different than B? So, that's actually what we're looking for ahead of time is like these have this slightly different signature. We

13:40don't know if it's actually a good pattern that we want to force more into and we want to like maybe develop a specific agentic skill to attack this problem because it's so interesting, or it could be this anti behavior where it's lazy or whatever it may be because it's told to conserve resources somewhere in the system prompt. And so, just discovering it's different is that first step. After that, then we

14:02can use LLMs to say, why is it different? Is this a good difference or a bad difference? And it turns out these reasoning models are actually pretty good at saying like, hey, looking at the macro level and being able to see all of the actual like the full trace, I can say this probably isn't what was intended and then we can come up with a suggestion of like, hey,

14:23put this in Claude code to like write a pull request to like automatically try to like run an experiment to see if it if it goes away. Um it reminds me of like back in the old school ML days like some when we used to talk like how some of the the way that you would use evals or or objective function reward functions for these systems, you

14:44could only really learn with experience. And there's like classic examples with fraud detection where it was like, maybe naively you say, oh, I want just the highest accuracy fraud detection. And then it's like, oh, wait a minute, actually precision and recall both matter. And then it's like, wait a minute, I could have the highest possible F1 score ever, but if the one transaction that's fraudulent that gets through was

15:07for a billion dollars, that kills my entire business. So, now I need to like incorporate magnitude into this. And then like time and geo and you end up creating this relatively complex way to both train at the time which were more traditional ML systems, but also monitor these systems. But that analysis was data science. That was a human looking through individual things of misclassification, true and false positive, trying to find these