In self-attention, each token generates a query, key, and value vector, and the model compares every query against every key to determine how much each token's meaning should be shaped by the other tokens around it.

33:06measuring error, and nudging every weight in the direction that reduces it. Now, if you zoom in on a block, self-attention is the core operation. For each token, the model generates three vectors from its embedding: a query, a key, and a value. Think of the query as what this token is looking for. The key is what each token offers, and the value is what gets passed along when

33:26there's a match. The model compares every query against every key, scores the matches, and mixes the values accordingly. The output is a new vector for each token, the same shape as the input, but now carrying information from the rest of the sentence. So, the word bank in the sentence "Bank of the river was muddy" ends up with a vector shaped mostly by river and muddy. Whereas, the word bank in the sentence

33:51"I went to the bank to deposit money" ends up shaped by deposit and money. So, the same word in, different vector out, meaning resolved by context. Then, multi-head attention runs that operation many times in parallel, with a different set of query, key, and value weights each time. Each run is a head, and nobody tells the head what to focus on. They start random, and during training, they end up specializing. One head might

34:14learn to track which pronoun refers to which noun, another might track verb tenses, another might track position. It's emergent and not designed. >> [music] >> And those outputs get combined back into a single vector per token. And feed forward layers come next. Each token's vector, now context enriched from attention, passes through a small two-layer network. The feed-forward step processes each token on its own, refining the signal before it moves on.

34:37And finally, you stack these layers. So, attention plus feed forward is one transformer block or one layer. And modern LLMs stack dozens or hundreds of these layers. And every block is the same input and output shape, one vector per token. Each layer builds on the last. So, early layers tend to capture syntax, middle layers capture meaning, late layers capture reasoning. And that's what deep learning means. It's

35:02mini stacked layers. So, that's the whole architecture. Tokens come in, get embedded, get passed through stacks of attention and feed forward, and a prediction comes out the other end. Everything else, whether it's GPT, Claude, or Gemini, is the same recipe just scaled up. So, let's take a look at how I implemented this to train my own simple transformer model. Okay, this next demo is called train transformer, and the arguments we pass