SambaNova's RDU architecture fuses an entire model decode step into a single hardware-orchestrated kernel with a technique called kernel looping, keeping HBM bandwidth continuously utilized instead of the stop-start kernel-by-kernel data movement that limits GPUs.
Kunle Olukotun explains that SambaNova's RDU chips collapse the whole decoder into one continuously-looping kernel, avoiding the repeated HBM round-trips and kernel-launch overhead that waste bandwidth on GPUs. ✦ AI generated
Kunle Olukotun · The Cognitive Revolution · 2026-07-02 · original ↗
starts at this moment · 116:20
“Could you contextualize a little bit — let's say you have a llama 3 model and you have a normal Nvidia H100 or B200 chip versus an RDU chip — what is the difference in the amount of HBM required?”
the way that things work on a RDU in a data flow is essentially you take the decoder and you make that a single kernel, right? And then you go even further and you use a technique that we've developed called kernel looping.
verbatim transcript · starts at 116:20
116:20called kernel looping. Whereas because you've got a single kernel and you need now if for instance if you don't thinking about llama 370B you have to run that uh uh decoder 32 times well you keep that single kernel decoder on the array of chips at the same time uh and then you just keep looping right and so the net result is you keep the HBM completely occupied
116:47>> right >> and you don't ever send any intermediate data between the kernels across the GPU or the RDU HBM boundary. Right? So you have both a more efficient use of the HBM bandwidth and you have a more complete use of it of the bandwidth. But we're not done there yet because the key innovation and I kind of alluded to it early earlier is that you've just you
117:12know because you're running across multiple chips and you're using what we call tensor level parallelism, right? So at some point you now need to gather all those results together, right? In an all reduce. >> Mhm. >> That's communication, right? You don't want to have that communication uh be a thing that limits that that stops the pipeline, >> right? So what we're able to do is we're able to communicate from one RDU chips
117:42SRAMM to another RDU chips SRAMM without going through HBM. This is called you know we terminate the communication uh inside the S SRAMM right so we don't use HBM bandwidth and more importantly it means that we could just treat the communication as another pipeline stage that we overlap with all the other kernel uh uh components of the uh decode algorithm right and so we get this more
118:13effective use of the HBM bandwidth we keep the HBM >> uh uh running. We keep the HPM utilized all the time and we go go back to our me that metric that we talked about uh memory bandwidth utilization, right? This is how we push it as close as possible to one, right? uh because we uh make sure that we only move the data that we absolutely have to move from HBM
- ·Entire decoder step fused into a single kernel
- ·Hardware-orchestrated, not kernel-by-kernel like GPUs
- ·Uses 'kernel looping' technique from SambaNova
- ·Avoids repeated HBM round-trips per kernel launch
- ·Keeps HBM bandwidth continuously utilized
- ·GPUs waste bandwidth on kernel-launch overhead
- ·GPUs waste bandwidth on repeated HBM round-trips
- ·Kernel-launch overhead limits GPU decode efficiency
- ·RDU's data-flow design sidesteps both bottlenecks