LLMs generate text autoregressively, predicting and appending one token at a time with no overall plan for how the output will end.
CJ explains that after producing one token, the model appends it and reruns the entire pipeline to predict the next, meaning coherent-looking paragraphs are actually built blindly, one step at a time. ✦ AI generated
CJ · Syntax · 2026-07-10 · original ↗
starts at this moment · 42:44
To get the next token, we append the token to the input and then run this entire process again so we get new attention, new distribution, new predicted token, append, repeat. And this whole process is known as auto regressive generation. Essentially, the model has no plan. It doesn't know how the sentence ends when it starts. So, every coherent paragraph from an LLM emerges one blind step at a time.
verbatim transcript · starts at 42:44
42:44tokens are considered. It essentially shrinks down the possible list of next tokens. >> [music] >> And those two parameters are how we can get different outputs from the exact same input prompt. Now, everything we've walked through so far, tokenization, embeddings, the transformer layer, softmax sampling, produces one single token. To get the next token, we append the token to the input and then run this entire process again so we get new attention, new
43:09distribution, new predicted token, append, repeat. And this whole process is known as auto regressive generation. Essentially, the model has no plan. It doesn't know how the sentence ends when it starts. So, every coherent paragraph from an LLM emerges one blind step at a time. Now, when you send your prompt off to a production LLM like ChatGPT or Claude or Gemini, it doesn't see just your prompt. It actually sees a structured package
43:36that includes the system prompt, which is essentially hidden instructions written by the AI provider themselves, things like you're a useful chatbot. And it also contains [music] the full conversation history. So, every single message that you've sent in that chat so far. And then finally, your latest prompt or your latest message at the end. So, the model has zero memory between requests. The conversation history is the actual memory of that
43:59conversation and that's what feeds into the auto regressive loop. So, with that in mind, your prompt in the history and everything that's gone into your current conversation, has to fit into a specific token limit known as the context window. And we saw in our simple transformer example, it had a context window of 32 tokens. But modern models have context windows anywhere from 100,000 to millions of tokens. And
44:20essentially, the context window is all the transformer can see when it's producing that next token. So, that's why when you have a long conversation, the answers from an LLM might start to degrade, or you might actually start to see hallucinations because with a longer context, there's more things the model needs [music] to look at when predicting the next token. So, it might ignore previous instructions because maybe later
44:40messages have more weight in the overall attention of your chat history. Now, at this point, we really have the full picture. Your prompt gets broken down into tokens. Those tokens are according to some vocabulary that was predefined based on the model's training data ahead of time. Then, those tokens get turned into embeddings, which actually give the overall prompt meaning. Then, the embedded tokens are passed through the transformer.