The real bottleneck in AI inference isn't raw compute (FLOPs) but memory bandwidth — moving weights and the KV cache to the compute units — and winning that bottleneck is what lets you generate 'premium tokens': fast, accurate output from large models.

102:21utilization the MFU. >> Um what what is your northstar as you guys are designing the chip like is that is that is that the strongest consideration or are some other metrics or numbers things that you focus on? No, the speed of light of of doing inference is really uh especially if you want high speed inference, right? So, everybody, you know, knows that we're in the aentic envir era and and so what one wants is

102:50what we call premium tokens, right? So, premium tokens are tokens that you can charge the most money for because they are premium. And why are they premium? because they are they come from from very large models. So they're accurate, but they also are provided to you at high speed so that that they're they're useful from the point of view of of an agentic environment that needs many

103:14turns through the models and and potentially has vulnerable models that that interact, right? And so the question is, you know, uh how do you provide those premium tokens? And you provide those premium tokens by having very fast inference. And that is not going to be limited by floating point. It's going to be limited by moving the KV cache and the parameters from the uh HBM memory to the the the compute chip,

103:42right? And so we like to think of this as memory bandwidth utilization, right? So the speed of light is is one, right? is that you you use your memory bandwidth completely uh for one thing >> or maybe two things. Moving KB cache, moving uh uh the the the parameters from the the uh HPM to the the uh the compute uh unit uh so that you generate when you

104:13generate each token, right? And so in fact in many instances you're underus underutilizing the flops on the uh on the uh the GPU or the compute units because in fact you are running a memory memory bound problem right and in fact most a lot of of of the the problems uh that one runs on GPUs and especially inference are in fact not computebound. They are memory bound