SambaNova's dataflow architecture pushes memory bandwidth utilization to 70-80% of peak by orchestrating data movement in hardware, compared to the roughly 10-20% GPUs typically achieve because they synchronize data movement in software.
Kunle Olukotun explains that SambaNova's core technical advantage is hardware-orchestrated data movement that keeps memory bandwidth utilization near 70-80%, versus GPUs' typical 10-20%, since inference is fundamentally a data-movement problem, not a compute problem. ✦ AI generated
Kunle Olukotun · The Cognitive Revolution · 2026-07-02 · original ↗
starts at this moment · 101:20
“It strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than with through NVLink and uh advanced HPM integration and uh software optimizations like Tensor MT VLM. So is it is it are they actually trying to brute force their way into this?”
whereas uh GPU views are often running at you know maybe 10 to 20% of the uh capabilities of the resources right uh the bandwidth and and and and the uh the memory bandwidth and and and the communication resources our goal in a Samanova system is to push that to be 70 to 80% of of the peak.
verbatim transcript · starts at 101:20
101:20right uh the bandwidth and and and and the uh the memory bandwidth and and and the communication resources our goal in a Samanova system is to push that to be 70 to 80% of of the peak and so the idea is yeah everybody Everybody wants more capabilities from the underlying uh resources, but the key is keeping that those resources as effectively used as possible. And of course that gives you
101:50more benefit for for the uh uh the cost that you spend on on on on providing uh higher memory bandwidth with the latest HPM and and uh higher signaling uh frequencies and and communication bandwidths between the chips using the latest uh uh variety of of and vlink and so >> for a lot of the chip users they uh they often focus on this number the model flops
102:21utilization the MFU. >> Um what what is your northstar as you guys are designing the chip like is that is that is that the strongest consideration or are some other metrics or numbers things that you focus on? No, the speed of light of of doing inference is really uh especially if you want high speed inference, right? So, everybody, you know, knows that we're in the aentic envir era and and so what one wants is
102:50what we call premium tokens, right? So, premium tokens are tokens that you can charge the most money for because they are premium. And why are they premium? because they are they come from from very large models. So they're accurate, but they also are provided to you at high speed so that that they're they're useful from the point of view of of an agentic environment that needs many
103:14turns through the models and and potentially has vulnerable models that that interact, right? And so the question is, you know, uh how do you provide those premium tokens? And you provide those premium tokens by having very fast inference. And that is not going to be limited by floating point. It's going to be limited by moving the KV cache and the parameters from the uh HBM memory to the the the compute chip,
- ·Dataflow architecture hits 70-80% of peak bandwidth
- ·GPUs typically reach only 10-20% utilization
- ·Hardware orchestrates data movement, not software
- ·Inference is fundamentally a data-movement problem
- ·GPUs synchronize data movement via software
- ·SambaNova orchestrates movement directly in hardware
- ·Bandwidth, not compute, is the real bottleneck