Video · 2026-04-30 · 54m · 6 moments

How to Engineer AI Inference Systems [Philip Kiely] - 766

✦ AI generated

timeline · colored by role

Claim

Building an effective inference system requires broad expertise across many complex, disparate disciplines, similar to how a mixed martial artist must master multiple distinct fighting styles rather than excelling at just one.

Kiely uses an MMA analogy to explain that inference engineers must be competent across GPU programming, applied research (quantization, speculation, KV cache), and large-scale distributed systems all at once.

transcript

Philip Kiely: You need to have a wide range of expertise on a lot of very different complicated topics to build a truly effective inference system. There's what happens on the GPU, right? There's understanding CUDA level programming and PyTorch on top of that and then the inference engines on top of that.

provides context · 1

Claim◆

Inference has the fastest research-to-production timeline of any technical field, with new techniques going from paper to production support within hours.

Philip Kiely argues that inference engineering moves from research to production faster than any other field, sometimes in hours, citing an engineer who implemented a new quantization paper as a CUDA kernel just 31 hours after it was published.

transcript

Philip Kiely: Even within AI, what moves faster? Training ones, for example, if you want to train a model off of a new technique, it can still take weeks or months to fine-tune the hyper parameters and find the exact right way to sort of express that technique. But with inference, the timeline is often hours. A new model architecture comes out, you have to figure out how to support it day zero.

Mechanism◆

Inference performance characteristics like latency, throughput, and cost are adjustable trade-offs that engineers can tune, not fixed, immutable properties of a model as most AI engineers assume.

Kiely explains that treating latency, throughput, cost, and reliability as a tunable 'efficient frontier' rather than fixed API parameters lets engineers build features like priority queues or confidently quantize models for their specific product.

transcript

Philip Kiely: You can trade off between, for example, latency and throughput in a given inference engine by adjusting things as simple as batch size or things like, you know, adding or removing a speculation algorithm. When you do that, when you create sort of a spectrum of outcomes, a an efficient frontier of high-performance inference, then you start to understand like, wait, I can choose to change the way that I consume these systems.

Mechanism

Companies progress through a maturity cycle from relying entirely on per-token closed model APIs, to hyperscaler provisioned-throughput deals, to dedicated inference providers or in-house infrastructure as their AI product scales.

Kiely lays out a product-maturity progression (not company-size based) from starting on closed-API providers, to hyperscaler commits when cost or capacity problems hit, to dedicated inference providers or in-house/edge deployment.

transcript

Philip Kiely: Option number one is to rely entirely on per token closed model providers. And everyone starts here just about. Everyone should start here. It's really easy. You get frontier intelligence with an API key. And that's a hard thing to beat when you're starting out. I think the next level often looks like running into one of two problems, either a cost problem or a capacity problem.

extends · 1

Fact

Older GPU generations like Hopper retain strong demand and rising rental prices years after release, due to export-control-driven Chinese open-source optimization for Hopper, its FP8 quantization support, and its efficiency for smaller models.

Kiely explains that Hopper GPUs remain highly sought after even as Blackwell rolls out, partly because export controls push Chinese labs to optimize for Hopper, and because Hopper (unlike Ampere) supports FP8 quantization cost-effectively.

transcript

Philip Kiely: The thing is the time it takes for a GPU generation to first be manufactured and actually distributed to the point where everyone can get their hands on it is months to a year. And then there's another cycle of months to a year of porting all of the code in the industry to run on it. So actually Hopper GPUs in particular still are very very popular for inference.

Mechanism

The rise of agentic AI workloads, which make dozens to thousands of model requests per single user action, is what is driving the need for highly specialized, optimized inference systems.

Kiely argues that agents' fan-out of many sequential model calls per user action (versus one call in chat) makes speed and reliability of every individual model and connection critical, pushing the industry toward specialized inference.

transcript

Philip Kiely: I think that the demand for agents and multi-step inference is in fact what is driving the need for this very specialized inference optimization. In an agent world, when a user takes an action, you are making dozens, hundreds, maybe even thousands of requests, often times across different models. And if you want your agent to be fast and reliable, then every single one of these models needs to be fast and reliable.

supports · 1

Highlight slides

Inference Beats Every Field to Production✦ from: Inference has the fastest research-to-production timeline of any technical field, with new techniques going from paper to production support within hours.Paper to Production: Hours, Not Months✦ from: Inference has the fastest research-to-production timeline of any technical field, with new techniques going from paper to production support within hours.Inference Trade-offs Are Tunable, Not Fixed✦ from: Inference performance characteristics like latency, throughput, and cost are adjustable trade-offs that engineers can tune, not fixed, immutable properties of a model as most AI engineers assume.Why It Matters✦ from: Inference performance characteristics like latency, throughput, and cost are adjustable trade-offs that engineers can tune, not fixed, immutable properties of a model as most AI engineers assume.

Related episodes

Video · The Cognitive Revolution

Fable's Back, AI Engineer Recap, & SambaNova

→

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→

Video · a16z Podcast

Software in the Age of Agents | The a16z Show

→