Audio · 2026-08-03 · 5 moments

The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

Baseten just raised a $13B Series F and is now one of the leading kings of inference engineering. We go into everything you need to know for autoregressive and diffusion engineering. ✦ AI generated

Apple Podcasts ↗

GUEST Philip Kiely GUEST Ali Taha HOST Alessio Fanelli HOST Shawn Wang (swyx)

timeline · colored by role

Mechanism◆

Open-source video generation models trail closed models like Veo and Kling mainly because of the quadratic attention bottleneck from the sheer number of tokens, and long-form video will likely require autoregressive generation despite current quality shortcomings.

Ali explains the wide gap between open and closed video models, rooted in the token explosion and O(n²) attention cost — five seconds of video already means 35,000 tokens — and argues future long-form video will need autoregressive generation, even though every current autoregressive video model has poor quality.

transcript

Ali Taha: The video models are... The difference between the best open source LLM and best open closed-source LLM is very small... Video models are definitely not. There's a huge gap... difference is night and day. ... For attention, for just five seconds, you're running attention on 35,000 tokens... And because it's O(n²)... to generate a good cut scene of like one minute, it's almost impossible to do within the same compute time... Autoregressive video seems to me like that is the bet that the future's gonna be making, but there are no good open source autoregressive video models out there today... they have to be autoregressive in order to exceed that five second frame.

Mechanism◆

Most inference optimizations are lossless; the main lossy one is quantization, and the goal of inference engineering is getting faster while staying as close to 100% model fidelity as possible.

Philip explains how to preserve model quality while optimizing inference: KV caching and speculation are lossless, while quantization is the main lossy step, managed via data format, layer selection, and calibration to keep fidelity near 100%.

transcript

Philip Kiely: There's a few things on quality. Most inference optimizations are lossless. KV caching, for example. You are just recomputing or preventing recomputing the same values. Speculation, of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization. And that really comes down to, number one, data format, number two, which parts of the model you choose to quantize, which layers, and number three, like doing a lot of calibration on the quantized weights, to ensure that you're preserving all the outliers.

explains mechanism · 1provides context · 1rebuts · 1

Data◆

Quantizing more of a model can actually improve its fidelity, because quantization errors in different layers can cancel each other out.

Ali Taha describes research showing that quantizing more layers can yield better quality and 20% more throughput, by mathematically predicting which layers' quantization errors cancel, measured via KL divergence against the full-precision model.

transcript

Ali Taha: What Joshua showed in his mathematical proof where he had like a verifier in, is that you can predict which layers are going to have quantization errors that will cancel out with each other, and you choose to quantize those layers. And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider, so you get 20% more throughput of it because there's more layers than running an NVFP4, and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out, like one layer skewed to the right one layer skewed to the left, one layer skewed to the right. Your final logits distribution is more similar to the original distribution of the model, so you have better fidelity. And so the way we proved this was with KL divergence.

Prediction

Inference engineering still yields huge gains of 20%, 100%, or even 200%, and the race to 10x faster inference is realistic by stacking optimizations such as quantization, speculative decoding, and disaggregation.

Philip explains that unlike mature domains where gains are measured in basis points, inference still delivers dramatic speedups; combining quantization, a speculator, P/D disaggregation, and a tuned runtime can reach ~10x a naive 30-40 token/sec baseline.

transcript

Philip Kiely: We generally see the goal of, pushing to 10X that. But, not necessarily day zero, but by stacking enough optimizations, if you have, say like four optimizations, each of which doubles performance. Or sorry, three optimizations, each of which doubles performance, then you stack that up, that's an 8X gain... You are using the best hardware possible. You have a optimized speculator. You have done all of your quantization work. You are Seeing a pretty high cache hit rate. You are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput, but it is possible. So the spreads that you see... oftentimes can hit that range. 10X is of course very aggressive. It's oftentimes maybe more of a four to six times improvement.

explains mechanism · 1extends · 1gives example · 2

Prediction

Faster networking interconnect could unlock dramatically faster decoding, potentially near 100x, by eliminating the need to transfer KV cache through host memory to HBM between nodes.

Ali argues that the biggest future win for distributed inference is faster NICs, since KV cache currently must be staged through node and host memory into HBM; direct inter-node transfer could yield an order-of-magnitude speedup in decode.

transcript

Ali Taha: The answer is just faster next, like faster network chip communications. It seems to me that like more and more memory is the bottleneck... when you're doing serving at large, you have to transfer KV cache from one node to another. But the way that you do that is you tran- you find the KV cache, you find where it is, you transfer it to another node, you put it on that node's memory, and then you transfer it from that node's memory into the GPU... If you were to somehow be able to... have extremely fast NICs, you could, in theory, spare that HBM, and you could just transfer KV cache trans like directly from one node to another. This would give you like almost 100X speed up when you're doing this aggregated serving between nodes and nodes... If someone were to figure that out, it would literally be like a like two orders of magnitude faster to do decode. That would be my take.

provides context · 1

Highlight slides

Open vs. Closed Video Models: A Huge Gap✦ from: Open-source video generation models trail closed models like Veo and Kling mainly because of the quadratic attention bottleneck from the sheer number of tokens, and long-form video will likely require autoregressive generation despite current quality shortcomings.The O(n²) Token Bottleneck✦ from: Open-source video generation models trail closed models like Veo and Kling mainly because of the quadratic attention bottleneck from the sheer number of tokens, and long-form video will likely require autoregressive generation despite current quality shortcomings.Long-Form Video Needs Autoregressive Generation✦ from: Open-source video generation models trail closed models like Veo and Kling mainly because of the quadratic attention bottleneck from the sheer number of tokens, and long-form video will likely require autoregressive generation despite current quality shortcomings.Most inference optimizations are lossless✦ from: Most inference optimizations are lossless; the main lossy one is quantization, and the goal of inference engineering is getting faster while staying as close to 100% model fidelity as possible.Quantization fidelity depends on three knobs✦ from: Most inference optimizations are lossless; the main lossy one is quantization, and the goal of inference engineering is getting faster while staying as close to 100% model fidelity as possible.Quantization Errors Can Cancel Out✦ from: Quantizing more of a model can actually improve its fidelity, because quantization errors in different layers can cancel each other out.Results: 20% More, Better Quality✦ from: Quantizing more of a model can actually improve its fidelity, because quantization errors in different layers can cancel each other out.

Related episodes

Video · Syntax

I Built an LLM From Scratch

→

Audio · We Study Billionaires

RWH070: Hunting For Hidden Treasures w/ Christopher Begg

→

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→