Kunle Olukotun explains SambaNova's core architectural thesis: unlike training, inference is bottlenecked by moving weights and KV cache between memory and compute units, not by matrix-multiplication compute, which is why GPUs (optimized for training-era matmul) are inefficient at it. ✦ AI generated
Kunle Olukotun · The Cognitive Revolution · 2026-07-02 · original ↗
starts at this moment · 98:22
“Can you tell us a little bit about how Samanova uh has this different paradigm that addresses kind of the uh issues with GPUs and brings down the total cost of uh ownership for enterprises?”
the inference problem is not really a compute problem because as the models get bigger, you now need to move the weights and of course what we call the the KV cache, you know, into uh the compute units. And that is essentially a data movement problem, right? And it's a data movement problem, you know, from the memory to the uh compute units.
verbatim transcript · starts at 98:22
98:22very large matrix multiplication ations. And so what happened is over time uh GPUs put more and more of their silicon area into uh the exercise of of of of making these matrix multiplication uh uh capabilities better you know using the tensor cores. Uh but once you've trained a model right and you train a model once you now need to use that model of course and that's the inference problem. And
98:49the inference problem is not really a compute problem because as the models get bigger, you now need to move the weights and of course what we call the the KV cache, you know, into uh the compute units. And that is essentially a data movement problem, right? And it's a data movement problem, you know, from the memory to the uh compute units. And it's a data movement problem from uh you
99:14know, your your your your chip uh compute unit. And of course you need to scale uh to multiple chips in order to handle uh the computational requirements for very especially for very low uh latency high-speed inference. And so our focus was how do you design an architecture that minimizes the overhead of compute of of of communication and make sure that you can uh most efficiently use the core resource in the
99:50uh in the system which is the memory. And it's the memory. Memory isn't just one thing as you all know. It's a hierarchy of memories, right? And so the key thing is how do you orchestrate that hierarchy? How do you orchestrate the communication such that you keep everything as efficiently utilized as possible? And if you do it right, you can get a 5 to 10x improvement over
100:12where where GPUs are today. It it strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than with through NVLink and uh advanced HPM integration and uh software optimizations like Tensor MT VLM. >> Yeah. >> So is it is it are they actually trying to brute force their way into this? >> Yeah. I mean this you you really of course want to continue to get improved