The page that shares no words
An embedding is a list of numbers that stands for the meaning of a piece of text. Texts with similar meanings get lists that point the same way, so a program can compare meanings with arithmetic.
A library visitor searches for "borrowing rules". The page they need is headed "Lending policy". The two share no words, so a search that matches words finds nothing. Numbers can find it.
By the end of this lesson
Note
I can say what an embedding is and compare two with cosine similarity.
Key words
| Word | What it means | Example |
|---|---|---|
| Embedding | a list of numbers that stands for a meaning | |
| Vector | a list of numbers, seen as a point or an arrow | |
| Dot product | the sum of the products of matching numbers | |
| Cosine similarity | a score of how alike two arrows point | 0.98 |
| Retrieval | finding the pieces of text that fit a question |
The cells students fill in are left blank in this preview.
Real embeddings and our stand-ins
A real embedding has hundreds of numbers and is made by a model. Nothing in this module makes one.
The lessons use small stand-ins instead: invented lists of three numbers, and counts of words that a script works out itself. That way every step can be run and checked here. The method is the same.
A vector is a list of numbers
Think of a list of numbers as a point on a map. Two numbers are a point on a flat map. Three are a point in a room.
Give each place in the list a meaning. Here the three places are animal, vehicle and food. A big number means the word is strongly about that thing.
Worked example: four invented vectors
| Word | Animal | Vehicle | Food |
|---|---|---|---|
| cat | 0.9 | 0.1 | 0.1 |
| kitten | 0.8 | 0.1 | 0.2 |
| bus | 0.1 | 0.9 | 0.0 |
| scone | 0.1 | 0.0 | 0.9 |
The cells students fill in are left blank in this preview.
Cosine similarity scores how alike two vectors are
To compare two vectors you need three small steps. The dot product multiplies the numbers in matching places and adds them up. The length is the square root of a vector's dot product with itself.
The cosine is the dot product divided by the two lengths. It is 1 when the arrows point the same way, 0 at a right angle and below 0 when they point away.
Worked example: two small vectors
| Step | Working | Result |
|---|---|---|
| Dot product | 1 x 2 + 2 x 1 | 4 |
| Length of a | square root of (1 x 1 + 2 x 2) | 2.24 |
| Length of b | square root of (2 x 2 + 1 x 1) | 2.24 |
| Cosine | 4 divided by (2.24 x 2.24) | 0.8 |
The cells students fill in are left blank in this preview.
Write the maths as a loop
The next script holds the four vectors and writes the three steps as small functions. It prints the cosine of every pair.
It also builds a count vector: a fixed list of words, and how many times each appears in a sentence. Two sentences that use the same words get vectors that point the same way.
Write the vector script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Small made-up vectors: [animal, vehicle, food]. A real embedding has hundreds of numbers.' 'const vectors = {' ' cat: [0.9, 0.1, 0.1],' ' kitten: [0.8, 0.1, 0.2],' ' bus: [0.1, 0.9, 0.0],' ' scone: [0.1, 0.0, 0.9],' '};' 'function dot(a, b) {' ' let total = 0;' ' for (let i = 0; i < a.length; i++) total = total + a[i] * b[i];' ' return total;' '}' 'function length(a) { return Math.sqrt(dot(a, a)); }' 'function cosine(a, b) { return dot(a, b) / (length(a) * length(b)); }' > vec.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the vector script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const names = Object.keys(vectors);' 'for (let i = 0; i < names.length; i++) {' ' for (let j = i + 1; j < names.length; j++) {' ' const score = cosine(vectors[names[i]], vectors[names[j]]);' ' console.log(names[i] + \" and \" + names[j] + \": \" + score.toFixed(2));' ' }' '}' '// A count vector: how many times each word of a fixed list appears.' 'const vocabulary = [\"open\", \"late\", \"book\", \"fine\"];' 'function counts(sentence) {' ' const found = sentence.toLowerCase().split(\" \");' ' return vocabulary.map(function (word) {' ' return found.filter(function (one) { return one === word; }).length;' ' });' '}' >> vec.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the vector script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const one = counts(\"open late open\");' 'const two = counts(\"book open late\");' 'console.log(\"Counts:\", one.join(\" \"), \"and\", two.join(\" \"));' 'console.log(\"Cosine of the counts:\", cosine(one, two).toFixed(2));' >> vec.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it and read the scores
docker exec -w /home/node/project agent node vec.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What it prints
| Line | What it shows |
|---|---|
| cat and kitten: 0.99 | Very alike |
| cat and bus: 0.22 | Not alike |
| bus and scone: 0.01 | Almost nothing in common |
| Cosine of the counts: 0.77 | The sentences share the words open and late |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a cosine of 0.9 means "90 per cent the same". It does not. The score only ranks: 0.9 is closer than 0.5. Use it to put things in order, never as a percentage of truth.