How a language model reads text
Tokenisation is how a language model cuts text into pieces called tokens, and byte-pair encoding is the usual way to choose those pieces. Every word you type into a chatbot is cut up this way before the model sees it.
The pieces decide how much text fits in, how rare words are handled and what the model can spell.
By the end of this lesson
Note
I can turn a short text into tokens with byte-pair merges.
Key words
| Word | What it means | Example |
|---|---|---|
| Token | one piece of text the model handles as a unit | tea |
| Vocabulary | the list of every token the model knows | |
| Pair | two tokens that sit side by side | e a |
| Merge | joining a pair into one new token | |
| Byte-pair encoding | building tokens by merging the commonest pair, again and again | |
| Token id | the number that stands for a token |
The cells students fill in are left blank in this preview.
Words or characters: two simple ways to split
Simple text processing splits text into words. A language model needs a fixed vocabulary, and words make that hard. A new word, such as a name or a typo, is simply unknown.
Characters fix that, because a few dozen of them spell any word. But a sentence becomes very long, and long inputs cost more work.
Worked example: one text, two splits
| Split by | Tokens in "tea team meat team tea" | Vocabulary | Can it spell "teas"? |
|---|---|---|---|
| Words | 5 | 3: tea, team, meat | No, it is unknown |
| Characters | 18 | 4: t, e, a, m | Only with a new letter, s |
The cells students fill in are left blank in this preview.
Count both kinds of token
text = "tea team meat team tea"
words = text.split()
chars = []
for word in words:
for ch in word:
chars.append(ch)
word_vocab = []
for word in words:
if word not in word_vocab:
word_vocab.append(word)
char_vocab = []
for ch in chars:
if ch not in char_vocab:
char_vocab.append(ch)
print("word tokens:", len(words), "vocabulary:", word_vocab)
print("character tokens:", len(chars), "vocabulary:", char_vocab)
print("is teas a known word?", "teas" in word_vocab)
# Add the word steam to the text and press Run.
# Which vocabulary grows, and by how much?
Byte-pair encoding merges the commonest pair
Byte-pair encoding sits between words and characters. Start with characters. Count every pair of tokens that sit side by side inside a word. Merge the commonest pair into one new token and add it to the vocabulary.
Then count again and merge again. Each merge makes common words shorter. Rare words stay as smaller pieces, so nothing is ever unknown.
Worked example: counting the pairs
| Pair | Where it appears | Count |
|---|---|---|
| e a | tea, team, meat, team, tea | 5 |
| t e | tea, team, team, tea | 4 |
| a m | team, team | 2 |
| m e | meat | 1 |
| a t | meat | 1 |
The cells students fill in are left blank in this preview.
Count the pairs in the lab
docker exec jupyter python3 -c "from collections import Counter; words = 'tea team meat team tea'.split(); print(Counter(w[i:i + 2] for w in words for i in range(len(w) - 1)).most_common(3))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: three merges
| Merge | New token | tea | team | meat |
|---|---|---|---|---|
| Start | none | t e a | t e a m | m e a t |
| 1: e a (5) | ea | t ea | t ea m | m ea t |
| 2: t ea (4) | tea | tea | tea m | m ea t |
| 3: tea m (2) | team | tea | team | m ea t |
The cells students fill in are left blank in this preview.
Make the first merge
text = "tea team meat team tea"
words = []
for word in text.split():
words.append(list(word))
left = "e"
right = "a"
merged = []
for tokens in words:
out = []
i = 0
while i < len(tokens):
# Join the pair into one token, then jump past both.
if i + 1 < len(tokens) and tokens[i] == left and tokens[i + 1] == right:
out.append(left + right)
i = i + 2
else:
out.append(tokens[i])
i = i + 1
merged.append(out)
print(merged)
# Change the pair to "t" and "e" and press Run.
# How many tokens does the word meat have now?
A common mistake
Warning
Many people think a token is a word. With byte-pair encoding a common word is one token, but a rare word is several. So count tokens, not words, whenever you ask how much text fits into a model.
Each token becomes a number
The model never sees letters. Each token in the vocabulary gets a token id, its place in the list. After the three merges the vocabulary is t, e, a, m, ea, tea and team, numbered 0 to 6.
So "team meat tea" becomes team, m, ea, t, tea, which is the ids 6, 3, 4, 0, 5. Five tokens instead of eleven letters.
Turn tokens into ids
docker exec jupyter python3 -c "vocab = ['t', 'e', 'a', 'm', 'ea', 'tea', 'team']; print([vocab.index(t) for t in ['team', 'm', 'ea', 't', 'tea']])"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.