Video · 2026-07-10 · 52m · 6 moments

LLMs Explained: tokens, vectors, embeddings, transformers and more.

✦ AI generated

timeline · colored by role

Context◆

Language has deep statistical structure, and an LLM is fundamentally doing the same thing Claude Shannon measured in 1950 — predicting what comes next given enough context.

CJ traces the core idea of LLMs back to Claude Shannon's 1950 letter-prediction game, establishing that LLMs are statistical models of language that predict what comes next.

transcript

CJ: Claude Shannon, the father of information theory, sat down with his wife Betty to play a game. And he showed her a passage of text with the next letter hidden, and she had to guess what came next, letter by letter. And he measured how often she was right. But what he was really measuring is how predictable is written language. And his finding was, given enough context, the next letter is often nearly certain.

provides context · 1

Claim

The jump from Eliza to ChatGPT is not just a better chatbot — it is a completely different mechanism, because earlier chatbots were rule-based systems where every response had to be programmed in advance by a human.

CJ distinguishes rule-based chatbots (Eliza, Perry, ALICE, SmarterChild) from LLMs by mechanism, not just capability: one is hand-coded if-statements, the other is learned from data.

transcript

CJ: Every single one of these chatbots up until this point were based on rules or scripts or templates or decision trees. Essentially, a human had to sit down at a computer and program every single response in advance. So, the jump from Eliza to chat GPT isn't just a better chatbot, it's a completely different mechanism.

explains mechanism · 1

Mechanism

Tokenization via byte-pair encoding (BPE) is the first step in processing text for an LLM — it splits text into tokens (not words or characters) using a compression algorithm originally invented in 1994, and token count determines cost and context window limits.

CJ explains BPE tokenization in detail: how it was originally a data compression technique, how researchers repurposed it in 2015 for neural machine translation, and why token count matters for API pricing and context windows.

transcript

CJ: A token is the smallest unit of text a language model works with. The word the is one token. The word tokenization might be split into token and ization, two tokens. A space is often part of a token. A new line is a token. An emoji might be multiple tokens. The model doesn't see words the way you do, it sees tokens. And how does it decide where to split? It uses an algorithm called BPE or byte pair encoding.

explains mechanism · 1extends · 1provides context · 2

Mechanism

A neural network model is literally just a file containing weight values — numbers that were calculated through backpropagation training — and every neural network works this way, from a tiny XOR network to GPT.

CJ demonstrates that after training a neural network on XOR, the saved model file contains only a handful of numbers, and this scales directly to billion-parameter models.

transcript

CJ: After we train these neural networks, it actually creates a file in the dot data folder that contains the weights for that neural network. And so, when we talk about a model, it's literally just a file with weight values inside of it. So, for the single layer network, we're connecting those two inputs to one single output. So, we need a connection from the first input to the output and the second input to the output. And that connection has a weight, and that's why we see two weight values here.

explains mechanism · 2gives example · 1provides context · 1

Definition◆

The transformer architecture — attention plus feed-forward layers stacked repeatedly — is the engine behind every major LLM, and it was first described in the 2017 paper 'Attention Is All You Need'.

CJ explains how the transformer's self-attention mechanism lets each token gather context from every other token, resolves word ambiguity, and scales from a 52,000-parameter toy model to GPT's 120 billion parameters.

transcript

CJ: By 2016, Google's translation models were built on a type of neural network called an LSTM. They worked, but they were slow. Each word had to wait for the previous one to finish processing. To help with longer sentences, researchers had bolted on an add-on called attention. Now, in 2017, Jakub Uszkoreit at Google proposed dropping the sequential part entirely using only attention. And eight researchers built it, all equal contributors, and they titled the paper 'Attention Is All You Need.'

explains mechanism · 1provides context · 2

Prediction◆

LLMs have no true understanding — they are the most sophisticated pattern-matching autocomplete systems ever built, and their output is just statistical inference over text, not a replication of human intelligence.

CJ argues that LLMs are not genuinely intelligent in the human sense; they are alien intelligences — something entirely different from human cognition — because they are trained exclusively on written text, not embodied experience.

transcript

CJ: The more I feel like LLMs really just are the most sophisticated pattern matching autocomplete things we've ever built. I really don't think that they have true understanding. And I feel that the less magic there is, the less hand-waving there is in terms of trying to understand the underlying tech, the closer we'll get to really understanding how we can create better methods of working with LLMs.

explains mechanism · 1rebuts · 1supports · 1

Highlight slides

LLMs = Shannon's 1950 Prediction Game✦ from: Language has deep statistical structure, and an LLM is fundamentally doing the same thing Claude Shannon measured in 1950 — predicting what comes next given enough context.Language Has Deep Statistical Structure✦ from: Language has deep statistical structure, and an LLM is fundamentally doing the same thing Claude Shannon measured in 1950 — predicting what comes next given enough context.Transformer: The Architecture Behind Every Major LLM✦ from: The transformer architecture — attention plus feed-forward layers stacked repeatedly — is the engine behind every major LLM, and it was first described in the 2017 paper 'Attention Is All You Need'.Scaling: From 52K to 120B Parameters✦ from: The transformer architecture — attention plus feed-forward layers stacked repeatedly — is the engine behind every major LLM, and it was first described in the 2017 paper 'Attention Is All You Need'.LLMs: Pattern-Matching, Not Understanding✦ from: LLMs have no true understanding — they are the most sophisticated pattern-matching autocomplete systems ever built, and their output is just statistical inference over text, not a replication of human intelligence.The Alien Intelligence Thesis✦ from: LLMs have no true understanding — they are the most sophisticated pattern-matching autocomplete systems ever built, and their output is just statistical inference over text, not a replication of human intelligence.

Related episodes

Video · a16z Podcast

Is Defense the Next Trillion-Dollar Category? | a16z American Dynamism Summit

→

Video · The Cognitive Revolution

Exploring the J-Space

→

Video · Machine Learning Street Talk

Intelligence is collective, not artificial — Prof. Michael I. Jordan (UC Berkeley / Inria)

→