Analytics helps you discover which metrics to track in the first place — it reveals the unknown unknowns that you wouldn't know to monitor until you find them.
Using the analogy of a 'heat dome' as a newly relevant weather phenomenon, Clark explains that analytics uncovers the orthogonal signals you didn't know to look for, which then become trackable metrics in your monitoring system. ✦ AI generated
Scott Clark · The TWIML AI Podcast · 2026-05-07 · original ↗
starts at this moment · 45:44
“It's not that you can't see the important things as these independent numbers... it's that starting from blank slate, you might not realize that the tool call is the thing that you need to be tracking.”
analytics is how you learn that you should even look for that. And then you put it in... eventually everything is measurable and then you can put it into the monitoring system or whatever it may be. But finding those unknown unknowns, especially in a non-stationary environment, is a really difficult problem that needs to be automated with analytics is by hypothesis.
verbatim transcript · starts at 45:44
45:44like, okay, this part of the confusion matrix is actually worth billions of dollars, whereas the rest of it's kind of whatever. >> Got it. Got it. Got >> But, analytics is how you learn that you should even look for that. And then you put it in Yeah, eventually everything is measurable and then you can put it into the the the monitoring system or whatever it may be. But finding those
46:04unknown unknowns, especially in a non-stationary environment, is a really difficult problem that needs to be automated with analytics is is by hypothesis. >> And Distributional's tool is open source? >> Not open source, but open distribution. So, completely free to use, um deploys on prem, or you can use our SaaS offering for free. Um but a lot of our clients like to keep their logs cuz there's a lot of
46:31sensitive information in the logs. They like to keep that uh to themselves. So, it deploys on prem, uses whatever local or third-party LLM that you already have in place, um and runs these analytics alongside your uh monitoring system or like your nightly like ETL jobs or whatever it may be. >> If I'm using, you know, analytics broadly or, you know, Distributional's tool, like what is the what's the key metric that I'm
47:01tracking? Is it like improvement in you know, whatever metric that I care about like over time, or is there some uh you know, how do folks think about you know, metrics for applying optimization or or for analytics? >> Metrics for a metric discovery system. >> Exactly, yeah. Right right right right right. >> Let's go full meta. So, I would say uh is it is it finding things that you
47:32either wouldn't have found or would have taken you a long time to find? And then you could do some sort of calculation for like how much quicker am I finding it or what's the value of finding this thing um uh ahead of time. And so, it ends up looking closer to like how many like net new features or pull requests or new metrics discovered as opposed to this