I Built an LLM From Scratch

CJ: his finding was, given enough context, the next letter is often nearly certain. Now, this means that language has deep statistical structure, and over 70 years later, that's pretty much what an LLM is doing. It's predicting what is coming next.

CJ: LLMs are statistical models of language. They extract patterns and relationships from massive amounts of text, and they build a mathematical representation of how words relate to each other.

CJ: That breakthrough came in 1986. David Rumelhart, Geoffrey Hinton, and Ronald Williams published a paper in nature describing backpropagation, which is a method for training multi-layer networks. Essentially, you make a prediction, measure how wrong it is, and then propagate the error signal backward through every layer, adjusting each weight to make the prediction slightly less wrong.

CJ: David Rumelhart, Geoffrey Hinton, and Ronald Williams published a paper in nature describing backpropagation, which is a method for training multi-layer networks. Essentially, you make a prediction, measure how wrong it is, and then propagate the error signal backward through every layer, adjusting each weight to make the prediction slightly less wrong.

CJ: Now, a model breaks your prompt into pieces, not words, not characters, it's something in between. A token is the smallest unit of text a language model works with. The word the is one token. The word tokenization might be split into token and ization, two tokens.

CJ: BPE was invented in 1994 by Philip Gage as a data compression technique. It has nothing to do with language models... Instead of deciding in advance what all of the words are or what your vocabulary is, you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data.

CJ: Instead of deciding in advance what all of the words are or what your vocabulary is, you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data. So, common words like the stay whole and then rare words get split into sub word pieces.

CJ: For each token, the model generates three vectors from its embedding: a query, a key, and a value. Think of the query as what this token is looking for. The key is what each token offers, and the value is what gets passed along when there's a match. The model compares every query against every key, scores the matches, and mixes the values accordingly.

CJ: For each token, the model generates three vectors from its embedding: a query, a key, and a value. Think of the query as what this token is looking for. The key is what each token offers, and the value is what gets passed along when there's a match.

CJ: The model compares every query against every key, scores the matches, and mixes the values accordingly. The output is a new vector for each token, the same shape as the input, but now carrying information from the rest of the sentence. So, the word bank in the sentence "Bank of the river was muddy" ends up with a vector shaped mostly by river and muddy.

CJ: To get the next token, we append the token to the input and then run this entire process again so we get new attention, new distribution, new predicted token, append, repeat. And this whole process is known as auto regressive generation. Essentially, the model has no plan. It doesn't know how the sentence ends when it starts. So, every coherent paragraph from an LLM emerges one blind step at a time.

CJ: Essentially, real humans are hired to sit down and rate the responses of the model. Good chatbot-like responses get ranked high, bad responses get ranked low. Now, what defines good and bad is entirely up to the company that trains the model, and also up to the judgment of those human rankers.