AI inference is fundamentally a data-movement problem—moving model weights and the KV cache to compute units—rather than a raw compute/matrix-multiplication problem the way training is.

99:43the compute units. And that is essentially a data movement problem, right? And it's a data movement problem from the memory to the compute units. And it's a data movement problem from uh you know your chip uh compute unit and of course you need to scale to multiple chips in order to handle the computational requirements for very especially for very low latency high speeded inference. And so our focus was

100:13how do you design an architecture that minimizes the overhead of compute of communication and make sure that you can most efficiently use the core resource in the uh in the system which is the memory and it's the memory. Memory isn't just one thing as you all know. It's a hierarchy of memories, right? And so the key thing is how do you orchestrate that hierarchy? How do you orchestrate the

100:41communication such that you keep everything as efficiently utilized as possible? And if you do it right, you can get a 5 to 10x improvement over where where GPUs are today. >> It strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than with through NVLink and uh advanced HPM integration and uh software optimizations like Tensor MT, VLM. >> Yeah. >> So is it are they actually trying to

101:14brute force their way into this? Yeah, I mean this you you really of course want to continue to get improved improvements peak improvements on HBM bandwidth and uh chipto-chip communication by using the latest technology but then the key is how effectively do you use that bandwidth how effectively to use that communication and do you make sure that you don't waste it right and so whereas GPUs are often running at maybe 10 to