Why Your LLM Evals Are Missing Critical Failures

Scott Clark: mathematically, it's actually relatively straightforward to ask the question, is A different than B? So, that's actually what we're looking for ahead of time is like these have this slightly different signature... After that, then we can use LLMs to say, why is it different? Is this a good difference or a bad difference? And it turns out these reasoning models are actually pretty good at saying like, hey, looking at the macro level... I can say this probably isn't what was intended.

Scott Clark: the best way to come up with an Eval or set of evals that actually represents what you want and what you can optimize towards or automatically optimize towards is to have this recursive loop of continual re-refinement. And I think analytics is the way to take these larger patterns and signals and then use that to map down into a smaller like latent space or manifold of like these are the numbers I really care about.

Scott Clark: this non-stationarity, which is what you're pointing at here, is actually one of the strongest arguments for keeping this in an online loop. Because whatever evals and guardrails or whatever it was that worked before might not work tomorrow because the model may have shifted around it... It's like you're trying to box something into something in some high-dimensional box and it's going to find some dimension that maybe you didn't realize.

Scott Clark: analytics is how you learn that you should even look for that. And then you put it in... eventually everything is measurable and then you can put it into the monitoring system or whatever it may be. But finding those unknown unknowns, especially in a non-stationary environment, is a really difficult problem that needs to be automated with analytics is by hypothesis.