MechanismArticle
Xiaomi reached 1000 tokens/second on a 1-trillion-parameter model by codesigning the model with its software stack—FP4 quantization, DFlash speculative decoding, and TileRT-optimized inference—running on an ordinary 8-GPU commodity node rather than specialized hardware.
Xiaomi's MiMo-V2.5-Pro-UltraSpeed model hits 1000 tokens per second through co-designed quantization and speculative decoding techniques running on commodity 8-GPU nodes, reflecting a broader push by Chinese firms to maximize efficiency amid export controls. ✦ AI generated
Xiaomi · Import AI · 2026-06-15 · original ↗
Xiaomi was able to do this by codesigning the model with the software stack around it, including obvious things like FP4 quantization, as well as using DFlash (a "speculative decoding method based on block-level masked parallel prediction"), and also working closely with TileRT, software from startup Tile AI which speeds up LLM inference on commodity hardware.
Read full article ↗excerpt · fair-use quotation
Around this claim
This moment responds to
gives example → Efficient AI inference requires optimizing cost per token through memory technologies like KV cache and CXL, plus intelligent routing tools like Dynamo, matched to the right combination of GPU, CPU, and LPU.Kaushik Shirhatti · The TWIML AI Podcastgives example → Speculative decoding speeds up LLM inference by having a small draft model predict several tokens ahead, then having the large model verify them all in one batched pass — which is faster because batched verification is compute-bound rather than limited by memory bandwidth like standard one-token-at-a-time generation.Akshat Bubna · Latent Space