Diffusion models generate images by starting from pure noise and iteratively denoising them, and this denoising training objective is much more stable to optimize than the adversarial game used to train GANs.

4:06spending a lot of my time kind of like figuring out get these models to work even better. Take us back to the creation of diffusion models. Like, where did the inspiration come from? Yeah, so back then, uh the field was dominated by GANs, generative adversarial networks. And I you know, that's the that old approach where there's two neural networks. There is one that generates images and there's one that is uh kind

4:27of like trying to discriminate and figure out if they images are real or fake and then you train them one against each other and it's a very kind of like unstable and and challenging kind of optimization problem because they are, you know, there's this game-theoretic kind of aspect to it where you know, they need to outcompete each other, these two neural networks, and it was very very unstable, very very

4:49difficult to get it to work well. A lots of tricks were needed. And so, we were trying in my lab to experiment with alternatives. And one alternative was the usual auto-regressive approach where you kind of like generate the image uh let's say one pixel at a time and that's never worked particularly well for for for images and video and it still doesn't. Um it's just very slow and and

5:11uh not very accurate. And so, we came up with this alternative approach which is now called a diffusion model where essentially you generate an image by starting [clears throat] from noise and then iteratively refining it until you get a crisp kind of like nice image that, you know, is consistent with the prompt at the end. And the key benefit is that the training objective is very stable. The neural

5:37network is trying to just denoise an image. So, you take images, you add noise and you train a neural network to remove noise. It's a fairly standard, relatively easy kind of like optimization problem that you can use to train neural networks over large data sets and it works reasonably well. And um essentially, you can then use this neural network at inference time to generate images because the networks

6:00have been trained to remove noise, to improve uh the samples, to correct mistakes. And so, you can turns out that you can just start with pure noise and then you apply this denoising network a bunch of times and at the end, uh you get a really nice image. And the key benefit, I mean, back then people were not thinking about test time inference and uh those kind of things, but it has