Faster networking interconnect could unlock dramatically faster decoding, potentially near 100x, by eliminating the need to transfer KV cache through host memory to HBM between nodes.
Ali argues that the biggest future win for distributed inference is faster NICs, since KV cache currently must be staged through node and host memory into HBM; direct inter-node transfer could yield an order-of-magnitude speedup in decode. ✦ AI generated
Ali Taha · Latent Space · 2026-08-03 · original ↗
plays this moment only · 35:49 — 36:59
“Any other upcoming trends in inference engineering that we didn't cover?”
The answer is just faster next, like faster network chip communications. It seems to me that like more and more memory is the bottleneck... when you're doing serving at large, you have to transfer KV cache from one node to another. But the way that you do that is you tran- you find the KV cache, you find where it is, you transfer it to another node, you put it on that node's memory, and then you transfer it from that node's memory into the GPU... If you were to somehow be able to... have extremely fast NICs, you could, in theory, spare that HBM, and you could just transfer KV cache trans like directly from one node to another. This would give you like almost 100X speed up when you're doing this aggregated serving between nodes and nodes... If someone were to figure that out, it would literally be like a like two orders of magnitude faster to do decode. That would be my take.
verbatim transcript · starts at 35:49
Philip [01:33:58]: You can see it, that the world-- rest of the world doesn’t know about. The big ones are obvious. Models get bigger. Hardware gets more powerful. Users get used to a certain level of speed and demand a higher one. I think that some things I’m excited about are systems level. We still have a lot to think about in terms of composing multiple models together. If you think about a voice agent, there’s three to five models involved in that and the communication between those models. There’s a lot of new modalities that are coming out. There’s like the Cosmos, the new world model. There’s more research. Speech to speech is still like not entirely a thing, but it’s getting, it’s getting closer. There’s gonna be just a lot of new modalities to build around, which is gonna be exciting. And then, yeah, I think that the other thing to solve, which is something we’ve been solving for a long time and are not done with yet, is just going to be continuing to operate at another 10X scale as an industry. If you think about the degree of usage that AI has worldwide compared to, some of the more mature technologies both on consumer and business, it’s pretty clear that there could be multiple 10Xs more of demand. If you look at the infrastructure work industry-wide, it’s been stood up very quickly to meet a unprecedented spike in demand that is like not stopping. So yeah, there’s just a lot of problems to solve around like long tail reliability and, figuring out where we’re gonna get the next like 10X and 100X of tokens from.
Ali [01:35:49]: I’m gonna say, it’s gonna be a really boring answer, but I think the answer is just faster next, like faster network chip communications. It seems to me that like more and more memory is the bottleneck. You wanna have larger models. Right now, when you’re doing serving at large, you have to transfer KV cache from one node to another. But the way that you do that is you tran- you find the KV cache, you find where it is, you transfer it to another node, you put it on that node’s memory, and then you transfer it from that node’s memory into the GPU, and for like into the tensor cores of the GPU. So there’s like a stage transfer here that makes it such that you’re very bottlenecked with just KV cache transfers at large, which affects the time of decode and PD disagg. You have to do this because the HBM is so - it’s like extremely fast, like 4.5 terabytes per second as opposed to. Like, which is like magnitudes better than NIC communication speed. If you were to somehow be able to, in like this theoretical dreamland, have extremely fast NICs, you could, in theory, spare that HBM, and you could just transfer KV cache trans like directly from one node to another. This would give you like almost 100X speed up when you’re doing this aggregated serving between nodes and nodes. I’m not familiar with the technical challenges of making NICs faster. I’m certain there’s a reason why they’re like orders of magnitude
Ali [01:36:59]: Smaller, like slower than, like HBM. But if someone were to figure that out, it would literally be like a - like two orders of magnitude faster to do decode. That would be my take.
Philip [01:37:12]: Be a good trip.