A neural network model is literally just a file containing weight values — numbers that were calculated through backpropagation training — and every neural network works this way, from a tiny XOR network to GPT.
CJ demonstrates that after training a neural network on XOR, the saved model file contains only a handful of numbers, and this scales directly to billion-parameter models. ✦ AI generated
CJ · Syntax · 2026-07-10 · original ↗
starts at this moment · 18:31
After we train these neural networks, it actually creates a file in the dot data folder that contains the weights for that neural network. And so, when we talk about a model, it's literally just a file with weight values inside of it. So, for the single layer network, we're connecting those two inputs to one single output. So, we need a connection from the first input to the output and the second input to the output. And that connection has a weight, and that's why we see two weight values here.
verbatim transcript · starts at 18:31
18:23course, with a very simple training text, right? It's literally one sentence. It's going to find each word as a as a unique pair. But in the code, we actually can specify how many merges to do maximum. But if we change this to something like three and train it on the same small bit of text, you'll see that it actually finds AT as a unique token cuz at appears multiple time in the
18:45training data. So, cat sat on mat. Now, because the word the appears twice, it actually got its own token and then everything else were just extra letters added on. So, those appear as individual letter tokens. So, if we dive into the code, one of the first things to look at is this regular expression. And every tokenization algorithm first runs all of the training data through this regular
19:05expression to split it up because BPE never merges across word boundaries. So, even before we start doing these merges, we need to define ahead of time like what are the whole groups we're going to use to actually find the individual merged elements within them. And in this case, our groups are actually words. Um but if you look at the algorithm for GPT-2 or GPT-4, they have a predefined
19:25really complex regular expression cuz they've defined some rules up front of how to split things ahead of time. But our initial step here is to just split on whole words. So, the things that we're going to be looking at to actually merge are the individual words by splitting on spaces essentially. Now, the next step is actually an optimization, and that is we determine the frequency of all of those words. So,
19:44our regular expression splits on spaces, and then every single unique piece of text in there, we count the number of occurrences. And the number of occurrences is basically going to give us a weight as to how much we actually care about that token in the training data. So, in the simple sentence, "The cat sat on the mat," the appears twice, so it's going to have a higher weight
20:03than some of the other words that we're looking at. And then we get into our actual training algorithm. Now, from there, we're going to iterate up to our max number of merges. And like I showed earlier, we have this set to 10,000. But you essentially get to decide ahead of time how much iteration you want to do, and that's going to determine how large your vocabulary actually gets. And with
20:21a really large max merge size, we're more likely to find all of the unique tokens in in a given training set. And then we have the bulk of the work. So, this essentially looks at every character pair to find the most common occurring ones, and also takes into account the weight, so how often that particular word occurs. And so, we find the most common occurring pair that has