Backpropagation, published in 1986 by Rumelhart, Hinton, and Williams, works by making a prediction, measuring the error, and propagating that error backward through every layer to adjust weights, and this same algorithm still trains every modern neural network including today's LLMs.

9:39and mathematically impossible for a single layer perceptron. But, multi-layer networks could solve exclusive or. The problem was nobody knew how to train them effectively. And that breakthrough came in 1986. David Rumelhart, Geoffrey Hinton, and Ronald Williams published a paper in nature describing backpropagation, which is a method for training multi-layer networks. Essentially, you make a prediction, measure how wrong it is, and then propagate the error signal backward

10:05through every layer, adjusting each weight to make the prediction slightly less wrong. And then you repeat this billions of times. Now, every neural network you interact with today, whether it's Claude or ChatGPT or Gemini, is trained with a some variant of this backpropagation algorithm. And so, let me show you the code and I'm going to use the exact same problem that killed the perceptron, exclusive or. Okay, so this example here is called XOR

10:30neural net and it will train a neural network in real time to solve exclusive or. So, if I passed in single layer, this trains a single layer network and you can see that it doesn't reach a point even after 5,000 iterations where it has the correct output that we're expecting. But, if I passed in multi-layer, this does reach a point where given these inputs we're getting our expected output here. And you can

10:53see that even around 800 iterations, so after 800 iterations, our loss is extremely low and then our loss just gets lower and lower from there. And then we really reach a point where the loss isn't much more after 5,000 iterations. So, after all of those iterations, we have weight values for our neural network that given two inputs, either false and false, false true, true false, or true and true, we

11:14get the expected outputs. And you can see this isn't perfect, but we do round these values, right? This rounds down to zero, this rounds up to one, up to one, down to zero. Now, the main thing that I want you to see in the code is the fact that after we train these neural networks, it actually creates a file in the dot data folder that contains the

11:31weights for that neural network. And so, when we talk about a model, it's literally just a file with weight values inside of it. So, for the single layer network, we're connecting those two inputs to one single output. So, we need a connection from the first input to the output and the second input to the output. And that connection has a weight, and that's why we see two weight