Self-attention resolves a word's meaning by comparing it against every other token in the sentence, so the same word gets a different vector depending on its surrounding context.

33:06measuring error, and nudging every weight in the direction that reduces it. Now, if you zoom in on a block, self-attention is the core operation. For each token, the model generates three vectors from its embedding: a query, a key, and a value. Think of the query as what this token is looking for. The key is what each token offers, and the value is what gets passed along when

33:26there's a match. The model compares every query against every key, scores the matches, and mixes the values accordingly. The output is a new vector for each token, the same shape as the input, but now carrying information from the rest of the sentence. So, the word bank in the sentence "Bank of the river was muddy" ends up with a vector shaped mostly by river and muddy. Whereas, the word bank in the sentence

33:51"I went to the bank to deposit money" ends up shaped by deposit and money. So, the same word in, different vector out, meaning resolved by context. Then, multi-head attention runs that operation many times in parallel, with a different set of query, key, and value weights each time. Each run is a head, and nobody tells the head what to focus on. They start random, and during training, they end up specializing. One head might

34:14learn to track which pronoun refers to which noun, another might track verb tenses, another might track position. It's emergent and not designed. >> [music] >> And those outputs get combined back into a single vector per token. And feed forward layers come next. Each token's vector, now context enriched from attention, passes through a small two-layer network. The feed-forward step processes each token on its own, refining the signal before it moves on.

34:37And finally, you stack these layers. So, attention plus feed forward is one transformer block or one layer. And modern LLMs stack dozens or hundreds of these layers. And every block is the same input and output shape, one vector per token. Each layer builds on the last. So, early layers tend to capture syntax, middle layers capture meaning, late layers capture reasoning. And that's what deep learning means. It's

35:02mini stacked layers. So, that's the whole architecture. Tokens come in, get embedded, get passed through stacks of attention and feed forward, and a prediction comes out the other end. Everything else, whether it's GPT, Claude, or Gemini, is the same recipe just scaled up. So, let's take a look at how I implemented this to train my own simple transformer model. Okay, this next demo is called train transformer, and the arguments we pass

35:24in are the number of epochs, and then we have arguments here for the actual generation of the next token, so that's temperature and top P, which we'll talk about later. And then you have the number of layers in that transformer, and then the number of tokens to predict. So, transformers only predict one token, but we'll talk about in the next section how we can predict multiple. And we essentially created a transformer