From token ids to useful numbers
A token embedding is a row of numbers that stands for one token, and positional encoding adds a signal that says where the token sits. Together they are the input of every transformer.
A token id is only a label. The number 2 is not "more" than the number 1. The model needs numbers it can multiply and add.
By the end of this lesson
Note
I can build the input vectors of a transformer from token ids and positions.
Key words
| Word | What it means | Example |
|---|---|---|
| Vector | a list of numbers | [1, 0, 2] |
| Embedding table | a table with one row of numbers for each token | |
| Embedding | the row of the table for one token | |
| Position | where a token sits in the text, counting from 0 | |
| Position signal | a vector that stands for one position | |
| Input vector | the embedding plus the position signal |
The cells students fill in are left blank in this preview.
An embedding table turns an id into numbers
An embedding table has one row for each token in the vocabulary. To embed a token, take the row whose number is the token id. That row is the token's embedding.
Our table has four tokens and three numbers in each row. The numbers start random and change in training, so tokens used in similar ways end up with similar rows.
Worked example: a tiny embedding table
| Token | Id | Row of numbers |
|---|---|---|
| bus | 0 | 1, 0, 2 |
| is | 1 | 0, 1, 0 |
| late | 2 | 2, 1, 1 |
| early | 3 | 2, 1, -1 |
The cells students fill in are left blank in this preview.
Look up rows and add positions
# One row of numbers for each token id: bus, is, late, early.
table = [[1, 0, 2], [0, 1, 0], [2, 1, 1], [2, 1, -1]]
# One row for each position in the sentence: first, second, third.
positions = [[1, 1, 0], [0, 1, 1], [1, 0, 1]]
ids = [0, 1, 2] # bus is late
for place in range(len(ids)):
row = table[ids[place]]
signal = positions[place]
vector = []
for j in range(3):
vector.append(row[j] + signal[j])
print("id", ids[place], "row", row, "plus position", signal, "gives", vector)
# Change ids to [2, 1, 0], which is late is bus, and press Run.
# Which token's vector changes, and which stays the same?
How big a real table gets
docker exec jupyter python3 -c "print(50000 * 512)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Order matters, and the table cannot show it
"bus is late" and "late is bus" hold the same three tokens. A lookup gives each token the same row wherever it sits.
The step in the next lesson, attention, mixes the tokens' vectors together. On its own it treats them as a bag with no order. So without help, the model could not tell the two sentences apart.
Worked example: two orders, the same rows
| Sentence | First | Second | Third |
|---|---|---|---|
| bus is late | bus: 1, 0, 2 | is: 0, 1, 0 | late: 2, 1, 1 |
| late is bus | late: 2, 1, 1 | is: 0, 1, 0 | bus: 1, 0, 2 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a model knows word order because the tokens arrive in order. Attention treats them as a bag. The model knows order only when a position signal is added to each token before the first layer.
A position signal is added to each token
A second table has one row for each position: first, second, third. Add the row for a token's position to its embedding, number by number. The result is the input vector.
Now "bus" first and "bus" third get different input vectors. Some models learn the position table in training. Others use fixed wave patterns that need no learning.
Worked example: adding the position signals
| Place | Position signal | "bus is late" | "late is bus" |
|---|---|---|---|
| First | 1, 1, 0 | bus: 1+1, 0+1, 2+0 = 2, 1, 2 | late: 3, 2, 1 |
| Second | 0, 1, 1 | is: 0, 2, 1 | is: 0, 2, 1 |
| Third | 1, 0, 1 | late: 3, 1, 2 | bus: 1+1, 0+0, 2+1 = 2, 0, 3 |
The cells students fill in are left blank in this preview.
Embed both sentences with NumPy
docker exec jupyter python3 -c "import numpy as np; E = np.array([[1, 0, 2], [0, 1, 0], [2, 1, 1], [2, 1, -1]]); P = np.array([[1, 1, 0], [0, 1, 1], [1, 0, 1]]); print((E[[0, 1, 2]] + P).tolist()); print((E[[2, 1, 0]] + P).tolist())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.