Language has deep statistical structure, and an LLM is fundamentally doing the same thing Claude Shannon measured in 1950 — predicting what comes next given enough context.

0:00What happens when you type a prompt into an LLM like ChatGPT or Claude or Gemini and press enter? The answer, as it turns out, involves over 80 years of research, billions of dollars, and some of the most elegant [music] math humans have ever invented. Now, to answer this question, I built an LLM from scratch, and I'm going to walk you through that full journey from keystroke to stream response with

0:20working code at every step. Now, our story actually starts back in 1950. Claude Shannon, the father of information theory, sat down with his wife Betty to play a game. And he showed her a passage of text with the next letter hidden, and she had to guess what came next, letter by letter. And he measured how often she was right. But what he was really measuring is how

0:41predictable is written language. And his finding was, given enough context, the next letter is often nearly certain. Now, this means that language has deep statistical structure, and over 70 years later, that's pretty much what an LLM is doing. It's predicting what is coming next. And LLMs are statistical models of language. They extract patterns and relationships from massive amounts of text, and they build a mathematical representation of how words

1:07relate to each other. And people often describe them as next token predictors, and while that's true, there is a massive amount of machinery involved before you can get to the point of what comes next. And that's really what I wanted to understand. Right? We are increasingly interacting with these AI services on a daily basis, and whether it's at home or at work, and a lot of the details are hidden from us.

1:31And most people kind of just hand-wave over, "Oh, it predicts the next token." And And I wanted to dive deeper. I wanted to have a better understanding of all this stuff. And so, that's exactly what I'm going to break down in this video. You're going to walk away with a deeper understanding of how LLMs work. If that sounds good, let's dive in. My name is CJ. Welcome to Syntax.

1:54Now, before LLMs existed, we had chatbots of forms over the past 60-plus years. And this really all started back in 1950 when Alan Turing published his paper computing machinery and intelligence. And this is where he proposed the famous Turing test, which stated if a machine can converse indistinguishably from a human, can it think? And this really set the target for the next 70 years. Now, in 1966, Joseph Weizenbaum at MIT

2:18built a program called Eliza, which was essentially just a pattern matching program that mimicked a Rogerian therapist. It had no understanding whatsoever. Essentially, whatever the user typed, it would go through a series of if statements to determine how it should respond. And a program like this is known as a rule-based system. Essentially, a programmer has to go in and manually write out all the statements to detect