GPUs typically waste most of their available memory bandwidth during AI inference, using only 10-20% of capability, whereas SambaNova's dataflow architecture pushes utilization to 70-80%.
Kunle Olukotun explains that GPUs run at only 10-20% of their memory bandwidth capability during inference because they execute decode steps kernel-by-kernel with data shuttling overhead, while SambaNova's RDU architecture fuses the decode into a single kernel and overlaps communication with compute to hit 70-80% utilization. ✦ AI generated
Kunle Olukotun · The Cognitive Revolution · 2026-07-09 · original ↗
starts at this moment · 101:48
“So is it are they actually trying to brute force their way into this?”
whereas GPUs are often running at maybe 10 to 20% of the capabilities of the resources, right? Uh the bandwidth and and the uh the memory bandwidth and the communication resources. Our goal in a Samanova system is to push that to be 70 to 80% of the peak.
verbatim transcript · starts at 101:48
101:4820% of the capabilities of the resources, right? Uh the bandwidth and and the uh the memory bandwidth and the communication resources. Our goal in a Samanova system is to push that to be 70 to 80% of the peak. And so the idea is yeah everybody wants more capabilities from the underlying uh resources but the key is keeping that those resources as effectively used as possible. And of
102:20course that gives you more benefit for for the uh the cost that you spend on on on providing a higher memory bandwidth with the latest HBM and uh higher signaling frequencies and communication bandwidths between the chips using the latest uh variety of of NVLink and stuff. >> Could we zoom out and just ask you to kind of taxonomize the whole chip space if you would? I mean this is a big
102:49question but I think people are familiar with things like Cerebras which obviously has this giant chip and has like a ton of memory on chip. We've seen a number of instances where people are burning the transformer architecture directly into the silicon with varying degrees I think of flexibility still remaining as they pursue that strategy. And I guess I'm I'm curious as to how you see the kind of
103:18menu of big different strategies, the big different bets that people are making and then also how you see the strengths and weaknesses of each. >> Yeah, that's a really interesting question. I think maybe you can think about it along three different axes, right? So one axis is your flexibility versus specialization axis right uh you know extreme flexibility might be something like a CPU or to some extent
103:46maybe a GPU which is this instruction driven execution engine right and so can be pretty flexible but of course you always pay overhead for executing instructions right and you pay overhead in terms of silicon area and in terms of time right and then you know on the extreme of that access would be something that would be very specialized for a very spec for a specific uh um