AI glossary
Attention
Attention is the step in a transformer where each token scores the tokens it may see, turns the scores into weights, and takes a weighted mix of their values.
Every token gets three vectors from three learnt tables: a query, what it is looking for; a key, what it offers; and a value, what it passes on. A query times a key gives a score for how well the two match.
Softmax turns the scores into weights that are positive and add up to 1, and the new vector is each value multiplied by its weight, added up. In a language model a causal mask hides the later tokens, so a token never sees the next token it is learning to guess.
A block runs several heads side by side, each with its own tables, so different heads can pick up different patterns. Every token scores every token it can see, so the work grows with the square of the length, one reason a very long prompt is slow.
A common mistake is to think attention picks one token and ignores the rest. It takes a share of every token it can see; a high weight only means a bigger share.
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.