The rise of agentic AI workloads, which make dozens to thousands of model requests per single user action, is what is driving the need for highly specialized, optimized inference systems.
Kiely argues that agents' fan-out of many sequential model calls per user action (versus one call in chat) makes speed and reliability of every individual model and connection critical, pushing the industry toward specialized inference. ✦ AI generated
Philip Kiely · The TWIML AI Podcast · 2026-04-30 · original ↗
starts at this moment · 37:21
“Are these ideas at tension, or does it just kind of point to a particular direction for inference?”
I think that the demand for agents and multi-step inference is in fact what is driving the need for this very specialized inference optimization. In an agent world, when a user takes an action, you are making dozens, hundreds, maybe even thousands of requests, often times across different models. And if you want your agent to be fast and reliable, then every single one of these models needs to be fast and reliable.
verbatim transcript · starts at 37:21
37:21direction for inference? I think that the demand for agents and and multi-step inference is in fact what is driving the need for this very specialized inference optimization. In a chat world, you are making Every time a user takes an action, you are making one request to one model. In an agent world, when a user takes an action, you are making dozens, hundreds, maybe even thousands of requests, often
37:49times across different models. And if you want your agent to be fast and reliable, then every single one of these models, as well as every connection between them, needs to be fast and reliable. And so, that makes inference a more critical challenge. And it also means that having a system where you are able to very quickly optimize individual models and very quickly optimize individual workloads, perhaps with a
38:19sort of AI-accelerated programming paradigm, means that you can actually, you know, build and scale these agents much more reliably. Talk about the the multimodality aspect of that and the kind of disparate workload aspect of that. Like, I I think I'm asking two distinct questions here. One is like, you know, we've got, you know, multimodal models. Like, what is different and interesting about them from an inference perspective, if
38:52anything? Um the other is more trying to get at in the agentic world, you kind of spoke to calling, you know, this fan out of like requests to like different types of models. Um I don't know, this question is kind of occurring to me. Like, you also have like tools like is you know, have you thought about or to what degree are people like really focusing on like
39:19tool engineering or tool inference engineering? I I don't know if this makes sense. Um but uh I'm curious like it it it's probably a separate question of like uh tool-at-scale system optimization and engineering that maybe is out of scope of inference, but, you know, if not, I want to hear your thoughts on that. I I like to think that that most things are in scope. Like we want to take as