Most inference optimizations are lossless; the main lossy one is quantization, and the goal of inference engineering is getting faster while staying as close to 100% model fidelity as possible.
Philip explains how to preserve model quality while optimizing inference: KV caching and speculation are lossless, while quantization is the main lossy step, managed via data format, layer selection, and calibration to keep fidelity near 100%. ✦ AI generated
Philip Kiely · Latent Space · 2026-08-03 · original ↗
plays this moment only · 25:00 — 27:13
“How do you see keeping quality of the model? ... What goes into keeping quality the same across them, right? Sure, you can run benchmarks but, like, how do you determine how much quantization are there standards?”
There's a few things on quality. Most inference optimizations are lossless. KV caching, for example. You are just recomputing or preventing recomputing the same values. Speculation, of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization. And that really comes down to, number one, data format, number two, which parts of the model you choose to quantize, which layers, and number three, like doing a lot of calibration on the quantized weights, to ensure that you're preserving all the outliers.
verbatim transcript · starts at 25:00
Vibhu [00:25:22]: But, like, how do you determine how much quantization are there standards? What goes into
Philip [00:25:27]: There’s a few things on quality. Most inference optimizations are lossless. KV caching, for example. You are just recomputing or preventing recomputing the same values. Speculation, of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization. And that really comes down to, number one, data format, number two, which parts of the model you choose to quantize, which layers, and number three, like doing a lot of calibration on the quantized weights, to ensure that you’re preserving all the outliers. There’s other tricks that you can do, though. A big one is long context, ‘cause one thing you asked at, right at the beginning is, “Oh, what’s gonna happen if I send a 200,000 token request in?” So with a long input sequence, you need to, store a lot more information. You need to process a lot more tokens. And so even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length, and of course a full length one as well. Because if someone doesn’t need the full million token context, for example, you can get them better performance. I don’t know if that’s exactly like quality of the model. The way that I think about quality is to what degree are we faithfully serving the original model? If you think of a golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, 100% fidelity of the model.
Philip [00:27:13]: You can also, of course, think about quality from the training side and how do you push yourself past 100%. But when I think about purely inference optimizations, it’s getting faster while staying as close to that 100% fidelity mark as possible. And certainly our standard internally is that, like you should not be able to tell the difference between our API and a, official API. I think Kimi in particular does a good job of vendor benchmarking here
Ali [00:27:41]: Yes
- ·KV caching avoids recomputing the same values
- ·Speculation rejects wrong draft tokens, no fidelity loss
- ·Only quantization introduces fidelity loss
- ·Data format determines precision vs. throughput
- ·Choosing which layers to quantize controls impact
- ·Calibration on quantized weights preserves outliers