Byte pair encoding lets a model learn its vocabulary automatically by iteratively merging the most common character pairs in training data, rather than requiring word lists to be defined in advance, which lets one algorithm handle any language or code without language-specific rules.

16:45find the pair that appears most frequently, replace it with a new symbol, and repeat. Essentially, it compresses data by finding common patterns. Now, 21 years later, in 2015, researchers Rico Sennrich, Barry Haddow, and Alexander Birch at the University of Edinburgh realized this same algorithm was perfect for building vocabularies for neural machine translation. Instead of deciding in advance what all of the words are or what your vocabulary is,

17:11you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data. So, common words like the stay whole and then rare words get split into sub word pieces. And the beauty of this is that one algorithm handles English, Japanese, Python code, TypeScript code, emojis, all without language specific rules. And tokenization isn't just a pre-processing detail. It has real consequences, right? Token count

17:39determines cost. Every API call is priced per token. Token count determines what fits into the context and every model has a maximum context window, which is measured in tokens. And if you exceed it, something's going to get cut. So, let's take a look at the code for the BPE tokenization. All right, so this demo is called the basic tokenizer. And essentially, you can drop in some text

18:00[music] and then it will show you how it essentially broke that training text down into tokens. So, with the cat sat on the mat, you can see at each merge the token pairs that it came across. And then finally, we get our overall vocabulary. And so, this came across six unique tokens total. And the main idea with BPE is this pair merging. The most frequent pairs are merged. And of

18:23course, with a very simple training text, right? It's literally one sentence. It's going to find each word as a as a unique pair. But in the code, we actually can specify how many merges to do maximum. But if we change this to something like three and train it on the same small bit of text, you'll see that it actually finds AT as a unique token cuz at appears multiple time in the

18:45training data. So, cat sat on mat. Now, because the word the appears twice, it actually got its own token and then everything else were just extra letters added on. So, those appear as individual letter tokens. So, if we dive into the code, one of the first things to look at is this regular expression. And every tokenization algorithm first runs all of the training data through this regular