Given enough context, written language is highly predictable, and this same statistical predictability is what modern LLMs exploit when they predict the next token.

0:41predictable is written language. And his finding was, given enough context, the next letter is often nearly certain. Now, this means that language has deep statistical structure, and over 70 years later, that's pretty much what an LLM is doing. It's predicting what is coming next. And LLMs are statistical models of language. They extract patterns and relationships from massive amounts of text, and they build a mathematical representation of how words

1:07relate to each other. And people often describe them as next token predictors, and while that's true, there is a massive amount of machinery involved before you can get to the point of what comes next. And that's really what I wanted to understand. Right? We are increasingly interacting with these AI services on a daily basis, and whether it's at home or at work, and a lot of the details are hidden from us.

1:31And most people kind of just hand-wave over, "Oh, it predicts the next token." And And I wanted to dive deeper. I wanted to have a better understanding of all this stuff. And so, that's exactly what I'm going to break down in this video. You're going to walk away with a deeper understanding of how LLMs work. If that sounds good, let's dive in. My name is CJ. Welcome to Syntax.

1:54Now, before LLMs existed, we had chatbots of forms over the past 60-plus years. And this really all started back in 1950 when Alan Turing published his paper computing machinery and intelligence. And this is where he proposed the famous Turing test, which stated if a machine can converse indistinguishably from a human, can it think? And this really set the target for the next 70 years. Now, in 1966, Joseph Weizenbaum at MIT

2:18built a program called Eliza, which was essentially just a pattern matching program that mimicked a Rogerian therapist. It had no understanding whatsoever. Essentially, whatever the user typed, it would go through a series of if statements to determine how it should respond. And a program like this is known as a rule-based system. Essentially, a programmer has to go in and manually write out all the statements to detect

2:41based on what the user has typed how it should respond. So, it has no built-in understanding. And famously, Weizenbaum's secretary was using the program, and she actually asked him to leave the room so that she could talk to Eliza privately. And he later wrote, "I had not realized that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." And we're seeing this today,