AI glossary
Transformer
A transformer is the neural network design most language models are built on: a stack of identical blocks in which every token uses attention to take in what it needs from the others.
Text goes in as tokens, and each token becomes a vector from an embedding table, with a position signal added because attention on its own does not know the order. The vectors then pass through the blocks one after another.
Each block does two things: attention, where every token mixes in a share of the tokens it can see, and a small feed-forward step applied to each token on its own. Residual connections and normalisation keep the numbers steady as blocks are stacked.
At the end, the last vector becomes a score for every token in the vocabulary, and training nudges the tables until the right next token scores highest. A tiny model in a lesson and a large one have the same parts; the large one has more blocks and wider tables.
A common mistake is to picture a transformer as a store of sentences it has read. What it keeps are the numbers in its tables, which is why it can write a sentence nobody wrote before and can also state something false.
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.