Running a trained AI model (inference) is fundamentally a data-movement/memory-bandwidth problem, not a compute/matrix-multiplication problem, because it requires continuously moving weights and the KV cache to compute units.

98:49the inference problem is not really a compute problem because as the models get bigger, you now need to move the weights and of course what we call the the KV cache, you know, into uh the compute units. And that is essentially a data movement problem, right? And it's a data movement problem, you know, from the memory to the uh compute units. And it's a data movement problem from uh you

99:14know, your your your your chip uh compute unit. And of course you need to scale uh to multiple chips in order to handle uh the computational requirements for very especially for very low uh latency high-speed inference. And so our focus was how do you design an architecture that minimizes the overhead of compute of of of communication and make sure that you can uh most efficiently use the core resource in the

99:50uh in the system which is the memory. And it's the memory. Memory isn't just one thing as you all know. It's a hierarchy of memories, right? And so the key thing is how do you orchestrate that hierarchy? How do you orchestrate the communication such that you keep everything as efficiently utilized as possible? And if you do it right, you can get a 5 to 10x improvement over

100:12where where GPUs are today. It it strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than with through NVLink and uh advanced HPM integration and uh software optimizations like Tensor MT VLM. >> Yeah. >> So is it is it are they actually trying to brute force their way into this? >> Yeah. I mean this you you really of course want to continue to get improved

100:47improvements peak improvements on HPM bandwidth and uh chip-to-chip communication by using the latest technology but then the key is how effectively do you use that bandwidth how effectively do you use that communication and do you make sure that you don't waste it right and so uh you know whereas uh GPU views are often running at you know maybe 10 to 20% of the uh capabilities of the resources