Tokenization via byte-pair encoding (BPE) is the first step in processing text for an LLM — it splits text into tokens (not words or characters) using a compression algorithm originally invented in 1994, and token count determines cost and context window limits.
CJ explains BPE tokenization in detail: how it was originally a data compression technique, how researchers repurposed it in 2015 for neural machine translation, and why token count matters for API pricing and context windows. ✦ AI generated
CJ · Syntax · 2026-07-10 · original ↗
starts at this moment · 15:55
A token is the smallest unit of text a language model works with. The word the is one token. The word tokenization might be split into token and ization, two tokens. A space is often part of a token. A new line is a token. An emoji might be multiple tokens. The model doesn't see words the way you do, it sees tokens. And how does it decide where to split? It uses an algorithm called BPE or byte pair encoding.
verbatim transcript · starts at 15:55
15:56basically be turned into inputs for that neural network. And the first step in that process is known as tokenization. Now, a model breaks your prompt into pieces, not words, not characters, it's something in between. A token is the smallest unit of text a language model works with. The word the is one token. The word tokenization might be split into token and ization, two tokens. A space is often part of a token. A new
16:22line is a token. An emoji might be multiple tokens. The model doesn't see words the way you do, it sees tokens. And how does it decide where to split? It uses an algorithm called BPE or byte pair encoding. And BPE was invented in 1994 by Philip Gage as a data compression technique. It has nothing to do with language models. And the idea was simple. Look at a sequence of bytes,
16:45find the pair that appears most frequently, replace it with a new symbol, and repeat. Essentially, it compresses data by finding common patterns. Now, 21 years later, in 2015, researchers Rico Sennrich, Barry Haddow, and Alexander Birch at the University of Edinburgh realized this same algorithm was perfect for building vocabularies for neural machine translation. Instead of deciding in advance what all of the words are or what your vocabulary is,
17:11you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data. So, common words like the stay whole and then rare words get split into sub word pieces. And the beauty of this is that one algorithm handles English, Japanese, Python code, TypeScript code, emojis, all without language specific rules. And tokenization isn't just a pre-processing detail. It has real consequences, right? Token count
17:39determines cost. Every API call is priced per token. Token count determines what fits into the context and every model has a maximum context window, which is measured in tokens. And if you exceed it, something's going to get cut. So, let's take a look at the code for the BPE tokenization. All right, so this demo is called the basic tokenizer. And essentially, you can drop in some text