A token, not a word or character, is the fundamental unit of text an LLM operates on, and it's produced by splitting text with the byte pair encoding (BPE) algorithm.
CJ explains that models break prompts into tokens (sub-word pieces found via byte pair encoding), not literal words or characters, and this choice directly affects cost and context window limits. ✦ AI generated
CJ · Syntax · 2026-07-10 · original ↗
starts at this moment · 15:56
Now, a model breaks your prompt into pieces, not words, not characters, it's something in between. A token is the smallest unit of text a language model works with. The word the is one token. The word tokenization might be split into token and ization, two tokens.
verbatim transcript · starts at 15:56
15:56basically be turned into inputs for that neural network. And the first step in that process is known as tokenization. Now, a model breaks your prompt into pieces, not words, not characters, it's something in between. A token is the smallest unit of text a language model works with. The word the is one token. The word tokenization might be split into token and ization, two tokens. A space is often part of a token. A new
16:22line is a token. An emoji might be multiple tokens. The model doesn't see words the way you do, it sees tokens. And how does it decide where to split? It uses an algorithm called BPE or byte pair encoding. And BPE was invented in 1994 by Philip Gage as a data compression technique. It has nothing to do with language models. And the idea was simple. Look at a sequence of bytes,
16:45find the pair that appears most frequently, replace it with a new symbol, and repeat. Essentially, it compresses data by finding common patterns. Now, 21 years later, in 2015, researchers Rico Sennrich, Barry Haddow, and Alexander Birch at the University of Edinburgh realized this same algorithm was perfect for building vocabularies for neural machine translation. Instead of deciding in advance what all of the words are or what your vocabulary is,
17:11you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data. So, common words like the stay whole and then rare words get split into sub word pieces. And the beauty of this is that one algorithm handles English, Japanese, Python code, TypeScript code, emojis, all without language specific rules. And tokenization isn't just a pre-processing detail. It has real consequences, right? Token count
17:39determines cost. Every API call is priced per token. Token count determines what fits into the context and every model has a maximum context window, which is measured in tokens. And if you exceed it, something's going to get cut. So, let's take a look at the code for the BPE tokenization. All right, so this demo is called the basic tokenizer. And essentially, you can drop in some text