AI inference is fundamentally a data-movement problem, not a compute problem — the bottleneck is moving model weights and the KV cache to compute units, not raw matrix-multiplication throughput.
Kunle Olukotun explains that unlike training, which is a compute-bound matrix-multiplication problem, inference is bottlenecked by moving weights and the KV cache between memory and compute units — the core insight behind SambaNova's chip architecture. ✦ AI generated
Kunle Olukotun · The Cognitive Revolution · 2026-07-09 · original ↗
starts at this moment · 99:19
“We asked how the company came to be.”
once you've trained a model, right, and you train a model once, you now need to use that model, of course, and that's the inference problem. And the inference problem is not really a compute problem because as the models get bigger, you now need to move the weights and of course what we call the KV cache into the compute units. And that is essentially a data movement problem, right?
verbatim transcript · starts at 99:19
99:19exercise of of making these matrix multiplication capabilities better using the tensor cores. But once you've trained a model, right, and you train a model once, you now need to use that model, of course, and that's the inference problem. And the inference problem is not really a compute problem because as the models get bigger, you now need to move the weights and of course what we call the KV cache into
99:43the compute units. And that is essentially a data movement problem, right? And it's a data movement problem from the memory to the compute units. And it's a data movement problem from uh you know your chip uh compute unit and of course you need to scale to multiple chips in order to handle the computational requirements for very especially for very low latency high speeded inference. And so our focus was
100:13how do you design an architecture that minimizes the overhead of compute of communication and make sure that you can most efficiently use the core resource in the uh in the system which is the memory and it's the memory. Memory isn't just one thing as you all know. It's a hierarchy of memories, right? And so the key thing is how do you orchestrate that hierarchy? How do you orchestrate the
100:41communication such that you keep everything as efficiently utilized as possible? And if you do it right, you can get a 5 to 10x improvement over where where GPUs are today. >> It strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than with through NVLink and uh advanced HPM integration and uh software optimizations like Tensor MT, VLM. >> Yeah. >> So is it are they actually trying to
101:14brute force their way into this? Yeah, I mean this you you really of course want to continue to get improved improvements peak improvements on HBM bandwidth and uh chipto-chip communication by using the latest technology but then the key is how effectively do you use that bandwidth how effectively to use that communication and do you make sure that you don't waste it right and so whereas GPUs are often running at maybe 10 to