Keys, Values and Queries

This is the second video on attention mechanisms. In the previous video we introduced self attention and in this video we're going to expand the idea by introducing keys, queries and values.

Video#

Links#

While making these videos we've found that these sources are very useful to have around. Not only because they help the conceptual understanding but also because some of them offer code examples.

• Transformers from Scratch
• The Illustrated Transformer
• Multi-Head Attention from d2l.ai

Exercises#

Try to answer the following questions to test your knowledge.

1. What parts of the self attention block are trainable when we have keys, queries and values?
2. If we want the shape of the output to be different as the input, what parts of the self attention block should we change?