CJ describes how BPE, invented in 1994 for data compression, was adapted in 2015 to build neural-network vocabularies by merging common character pairs, letting one algorithm handle any language without hand-written rules. ✦ AI generated
CJ · Syntax · 2026-07-10 · original ↗
starts at this moment · 16:22
BPE was invented in 1994 by Philip Gage as a data compression technique. It has nothing to do with language models... Instead of deciding in advance what all of the words are or what your vocabulary is, you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data.
verbatim transcript · starts at 16:22
16:22line is a token. An emoji might be multiple tokens. The model doesn't see words the way you do, it sees tokens. And how does it decide where to split? It uses an algorithm called BPE or byte pair encoding. And BPE was invented in 1994 by Philip Gage as a data compression technique. It has nothing to do with language models. And the idea was simple. Look at a sequence of bytes,
16:45find the pair that appears most frequently, replace it with a new symbol, and repeat. Essentially, it compresses data by finding common patterns. Now, 21 years later, in 2015, researchers Rico Sennrich, Barry Haddow, and Alexander Birch at the University of Edinburgh realized this same algorithm was perfect for building vocabularies for neural machine translation. Instead of deciding in advance what all of the words are or what your vocabulary is,
17:11you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data. So, common words like the stay whole and then rare words get split into sub word pieces. And the beauty of this is that one algorithm handles English, Japanese, Python code, TypeScript code, emojis, all without language specific rules. And tokenization isn't just a pre-processing detail. It has real consequences, right? Token count
17:39determines cost. Every API call is priced per token. Token count determines what fits into the context and every model has a maximum context window, which is measured in tokens. And if you exceed it, something's going to get cut. So, let's take a look at the code for the BPE tokenization. All right, so this demo is called the basic tokenizer. And essentially, you can drop in some text
18:00[music] and then it will show you how it essentially broke that training text down into tokens. So, with the cat sat on the mat, you can see at each merge the token pairs that it came across. And then finally, we get our overall vocabulary. And so, this came across six unique tokens total. And the main idea with BPE is this pair merging. The most frequent pairs are merged. And of