Word embedding vectors trained via word2vec capture real semantic relationships, like gender and royalty, purely from word co-occurrence statistics, without anyone explicitly teaching the model those concepts.

22:54neural network, an embedding model, to predict words from their neighbors in large amounts of text. It was a simple setup, but when they looked at the vectors the model produced, they discovered structure that no one had taught it. For instance, take the vector for the word king, subtract the vector for the word man, add the vector for the word woman, and the closest result is queen. Now, nobody told the model about gender

23:20or royalty. These concepts just emerged purely from the statistics of which words appear near other words. And this was essentially first idea, you know a word by it keeps implemented as math. Now, what is a vector? Essentially, it's a list of numbers that identifies a point in high-dimensional space. The simplest version is a vector in 3D space or three numbers, X, Y, and Z. However, embedding vectors are much

23:47larger. The word to vec paper used 300 dimensions, and GPT-3's the largest model uses 12,288 dimensions. Each number captures some feature the model learned during training, not something a human named, but together they encode meaning as a position in space. Words with similar meanings end up as nearby points. For instance, happy and joyful are close together, and happy and refrigerator are far apart. Now, we can measure exactly how close

24:16two vectors are using a formula called cosine similarity, which essentially looks at the angle between two vectors. A score of one means they're identical, a score of zero means they're completely unrelated. Now, modern LLM embeddings are direct descendants of word to vec, just scaled up from individual words to entire context. So, let's take a look at how to train a simple word to vec model. All right, this demo is called train

24:40embeddings, and you can pass in a couple of words, and it will train an embedding model in real time, and then show you the comparison of the generated vectors between these words that you pass in. And you can see for each of the words that we passed in, we see the actual generated vector. So, these are the embeddings that are generated for each of these tokens. But, the cool thing to