Word2vec embeddings trained purely on word co-occurrence statistics spontaneously encode concepts like gender and royalty as vector arithmetic, such that king minus man plus woman equals queen.

19:25really complex regular expression cuz they've defined some rules up front of how to split things ahead of time. But our initial step here is to just split on whole words. So, the things that we're going to be looking at to actually merge are the individual words by splitting on spaces essentially. Now, the next step is actually an optimization, and that is we determine the frequency of all of those words. So,

19:44our regular expression splits on spaces, and then every single unique piece of text in there, we count the number of occurrences. And the number of occurrences is basically going to give us a weight as to how much we actually care about that token in the training data. So, in the simple sentence, "The cat sat on the mat," the appears twice, so it's going to have a higher weight

20:03than some of the other words that we're looking at. And then we get into our actual training algorithm. Now, from there, we're going to iterate up to our max number of merges. And like I showed earlier, we have this set to 10,000. But you essentially get to decide ahead of time how much iteration you want to do, and that's going to determine how large your vocabulary actually gets. And with

20:21a really large max merge size, we're more likely to find all of the unique tokens in in a given training set. And then we have the bulk of the work. So, this essentially looks at every character pair to find the most common occurring ones, and also takes into account the weight, so how often that particular word occurs. And so, we find the most common occurring pair that has

20:42the highest weight, and that becomes a new piece that we're going to then merge on in the next iteration. So, we take that best pair that we found, merge all of the groups, in this case merge all of the words together, and then repeat to find the next most commonly occurring pair. So, to see a more interesting example, I'm going to plug the Bee Movie script into this tokenizer, and we can

21:01see it work to actually do all of the merges and find all of the unique tokens in the Bee Movie script. So, after the training, this found 2,088 unique tokens, which essentially is all of the unique words in the Bee Movie script. And so, you can see it found all the whole words. But, if we do reduce our max merges to something like 1,000, and then try this again, we are going to see

21:21tokens in our vocabulary that are essentially broken up words. Like you can see the word according got broken up into four tokens because we only have a certain token budget size, and the word according didn't appear that many times in the overall Bee Movie script. That's the basics of tokenizing. Okay. So, we've analyzed a large data set, extracted out all the possible tokens using this BPE algorithm, and