Inference engineering still yields huge gains of 20%, 100%, or even 200%, and the race to 10x faster inference is realistic by stacking optimizations such as quantization, speculative decoding, and disaggregation.

Philip explains that unlike mature domains where gains are measured in basis points, inference still delivers dramatic speedups; combining quantization, a speculator, P/D disaggregation, and a tuned runtime can reach ~10x a naive 30-40 token/sec baseline. ✦ AI generated

Philip Kiely · Latent Space · 2026-08-03 · original ↗

We generally see the goal of, pushing to 10X that. But, not necessarily day zero, but by stacking enough optimizations, if you have, say like four optimizations, each of which doubles performance. Or sorry, three optimizations, each of which doubles performance, then you stack that up, that's an 8X gain... You are using the best hardware possible. You have a optimized speculator. You have done all of your quantization work. You are Seeing a pretty high cache hit rate. You are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput, but it is possible. So the spreads that you see... oftentimes can hit that range. 10X is of course very aggressive. It's oftentimes maybe more of a four to six times improvement.

verbatim transcript · starts at 32:35