Kiely explains that treating latency, throughput, cost, and reliability as a tunable 'efficient frontier' rather than fixed API parameters lets engineers build features like priority queues or confidently quantize models for their specific product. ✦ AI generated
Philip Kiely · The TWIML AI Podcast · 2026-04-30 · original ↗
starts at this moment · 17:26
“What are the situations where I'm going to want to, you know, have deep knowledge about inference?”
You can trade off between, for example, latency and throughput in a given inference engine by adjusting things as simple as batch size or things like, you know, adding or removing a speculation algorithm. When you do that, when you create sort of a spectrum of outcomes, a an efficient frontier of high-performance inference, then you start to understand like, wait, I can choose to change the way that I consume these systems.
verbatim transcript · starts at 17:26
17:26these as sort of immutable discrete points and understanding them more as a spectrum. You can trade off between, for example, latency and throughput in a given inference engine by adjusting things as simple as batch size or things like, you know, adding or removing a speculation algorithm. When you when you do that, when you create sort of a spectrum of outcomes, a an efficient frontier of high-performance inference, then you
17:55start to understand like, wait, I can choose to change the the way that the way that I I the way that I consume these systems. So, for example, you might start having priority queues in your product where paid user traffic gets prioritized over free user traffic. And maybe that's something that you couldn't have built previously. Or maybe you add a concept of you know, maybe maybe you understand
18:25like I have the ability to quantize models and because I have my own very sophisticated and product-specific evals, I can do that with complete confidence that I'm not degrading the quality of the service my users are experiencing. And then when you quantize your own model and start to serve it faster and and less expensive and with a great deal of confidence that you've calibrated it properly, then you are not subject to sort of
18:54random intelligence degradations from a third-party provider who might not understand the exact nature of your workload. Instead, you've create you've achieved the sort of price and and performance goals that they're going for with that, but actually, you know, kept things appropriate for your task. So, you know, if you don't know a lot about inference, you might think like, oh, all quantized models are bad, when in fact,
19:19you can calibrate them appropriately to your workload. There's all sorts of, you know, examples like this. Even things as simple as, you know, understanding that the way KV cache reuse works means that you have to think about only your prefix and even one token difference early in a sequence can throw off the entire rest and and maybe that affects how you structure your chat template or how you