Standard GPUs typically use only 10-20% of their available memory-bandwidth and communication capability during inference, whereas SambaNova's dataflow architecture is designed to push effective utilization to 70-80% of peak.
Kunle Olukotun explains that SambaNova's reconfigurable dataflow chip is built to keep memory bandwidth near-fully utilized during inference, versus the 10-20% utilization typical of GPUs. ✦ AI generated
Kunle Olukotun · The Cognitive Revolution · 2026-07-02 · original ↗
starts at this moment · 100:47
“It strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than through NVLink and advanced HBM integration and software optimizations like TensorRT-LLM. So is it, are they actually trying to brute force their way into this?”
whereas uh GPU views are often running at you know maybe 10 to 20% of the uh capabilities of the resources right uh the bandwidth and and and and the uh the memory bandwidth and and and the communication resources our goal in a Samanova system is to push that to be 70 to 80% of of the peak
verbatim transcript · starts at 100:47
100:47improvements peak improvements on HPM bandwidth and uh chip-to-chip communication by using the latest technology but then the key is how effectively do you use that bandwidth how effectively do you use that communication and do you make sure that you don't waste it right and so uh you know whereas uh GPU views are often running at you know maybe 10 to 20% of the uh capabilities of the resources
101:20right uh the bandwidth and and and and the uh the memory bandwidth and and and the communication resources our goal in a Samanova system is to push that to be 70 to 80% of of the peak and so the idea is yeah everybody Everybody wants more capabilities from the underlying uh resources, but the key is keeping that those resources as effectively used as possible. And of course that gives you
101:50more benefit for for the uh uh the cost that you spend on on on on providing uh higher memory bandwidth with the latest HPM and and uh higher signaling uh frequencies and and communication bandwidths between the chips using the latest uh uh variety of of and vlink and so >> for a lot of the chip users they uh they often focus on this number the model flops
102:21utilization the MFU. >> Um what what is your northstar as you guys are designing the chip like is that is that is that the strongest consideration or are some other metrics or numbers things that you focus on? No, the speed of light of of doing inference is really uh especially if you want high speed inference, right? So, everybody, you know, knows that we're in the aentic envir era and and so what one wants is
- ·Standard GPUs run at only 10-20% of resource capability
- ·Applies to memory bandwidth and communication resources
- ·Most inference hardware sits mostly idle
- ·Reconfigurable dataflow chip targets near-full utilization
- ·Goal: push utilization to 70-80% of peak
- ·Design keeps memory bandwidth close to saturated during inference