Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free module

How language models work: transformers from scratch

Learn how a language model works from scratch: tokens, embeddings, attention, a transformer block, next-token training and sampling.

6 lessons370 min lessonAI and data science

The lessons

Each lesson is free to read in full. Create a free account to run its commands in a live lab and to sit its quizzes and tasks.

  1. 1 · Container lab

    Tokenisation and byte-pair encoding

    I can turn a short text into tokens with byte-pair merges.

    Read the lesson

  2. 2 · Container lab

    Token embeddings and positional encoding

    I can build the input vectors of a transformer from token ids and positions.

    Read the lesson

  3. 3 · Container lab

    How attention works in a transformer

    I can work out the output of attention for one token by hand.

    Read the lesson

  4. 4 · Container lab

    Inside a transformer block

    I can trace one token through the steps of a transformer block.

    Read the lesson

  5. 5 · Container lab

    Training a tiny language model

    I can train a tiny model to predict the next character and read its loss.

    Read the lesson

  6. 6 · Container lab

    Sampling, temperature and top-k

    I can choose the next token with greedy decoding, temperature and top-k.

    Read the lesson

Worked in a real lab

Every lesson of this module is worked in a lab started for you alone. The labs it uses:

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.