Keys, Queries, and Values: The celestial mechanics of attention

The attention mechanism is what makes Large Language Models like ChatGPT or DeepSeek talk well. But how does it work? One can see it as a mechanism that uses similarity to figure out what parts of the text to pay more or less attention to. For this, we use word embeddings. I like to see word embeddings as words flying around in the universe, like planets and stars. In this case, the attention mechanism (the Keys, Queries, and Values matrices) define the fabric of this universe, and the laws of gravity, that resemble (yet in some ways are very different) to the laws of gravity that rule our universe. Come join me in this celestial adventure in the universe of language! See other videos in this LLM series The attention mechanism in LLMs:    • The Attention Mechanism in Large Language ...   The math behind attention mechanisms:    • The math behind Attention: Keys, Queries, ...   Transformer models:    • What are Transformer Models and how do the...   Get the Grokking Machine Learning book! https://manning.com/books/grokking-ma... Discount code (40%): serranoyt (Use the discount code on checkout) 01:55 Similarity 02:12 Embeddings 04:56 Attention 07:14 Dot product 09:29 Cosine similarity 11:10 The Keys and Queries matrices 14:19 Compressing and stretching dimensions 18:50 Combining dimensions 23:14 Asymmetric pull 40:57 Multi-head attention 45:14 The Value matrix 49:24 Summary