It found something, but was it right?
Retrieval is finding the chunks that answer a question. Measuring it means asking questions whose answer you already know and counting how often the right chunk comes back.
A library site always shows three results. Staff say it feels good. Nobody has counted. Without a count you cannot tell whether a change made it better or worse.
By the end of this lesson
Note
I can find the chunks that answer a question and say how well I did.
Key words
| Word | What it means | Example |
|---|---|---|
| Retriever | the code that finds chunks for a question | |
| Hit | the right chunk came back | |
| Hit rate | the share of questions that were hits | 80 per cent |
| Test case | a question with a known source chunk | |
| Minimum score | the lowest score worth returning | 0.3 |
The cells students fill in are left blank in this preview.
A test case has a known source
Write down questions and, for each, the id of the chunk that holds the answer. Run the retriever on every question. A hit is when the right id is among the chunks it returns.
The hit rate is the number of hits divided by the number of questions. Four hits in five questions is a hit rate of 80 per cent.
Some questions have no answer
Visitors ask things the documents do not cover. For those, the right result is nothing. If the retriever returns its least bad chunk, a later step may build an answer from the wrong text.
A minimum score helps. Return a chunk only when its score is at least the minimum. Otherwise return an empty list.
Worked example: five test cases
| Question about | Expected | Retrieved | Result |
|---|---|---|---|
| Opening time | hours | hours | Hit |
| The fine | fines | rooms | Miss |
| The librarian | none | nothing | Hit |
| Parking | none | rooms | Miss |
| Booking a room | rooms | rooms | Hit |
The cells students fill in are left blank in this preview.
Score a retriever in a script
The next script rebuilds the library store from the vector store lesson, with the same count vectors. It adds a minimum score of 0.3 and returns at most one chunk.
It then runs five test cases, one of which has no answer, and prints each result and the hit rate.
Write the scoring script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const vocabulary = [\"open\", \"close\", \"saturday\", \"sunday\", \"borrow\", \"book\", \"late\", \"fine\", \"room\", \"library\", \"friday\", \"week\"];' 'const chunks = [' ' { id: \"hours-week\", text: \"The library opens at 9 am and closes at 6 pm from Monday to Friday.\" },' ' { id: \"hours-weekend\", text: \"On Saturday the library opens at 10 am and closes at 2 pm. It is closed on Sunday.\" },' ' { id: \"borrowing\", text: \"You can borrow 8 books for 3 weeks. You can renew a book twice online.\" },' ' { id: \"fines\", text: \"A late book costs 20p a day. The fine stops at 10 pounds.\" },' ' { id: \"rooms\", text: \"A study room holds 6 people. Book a room at the desk or online for 2 hours.\" },' '];' 'function stem(word) { return word.length > 3 && word.endsWith(\"s\") ? word.slice(0, -1) : word; }' > score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the scoring script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function vectorOf(text) {' ' const words = text.toLowerCase().replace(/[^a-z0-9 ]/g, \"\").split(\" \").map(stem);' ' return vocabulary.map(function (term) {' ' return words.filter(function (word) { return word === term; }).length;' ' });' '}' 'function cosine(a, b) {' ' let dot = 0, sa = 0, sb = 0;' ' for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; sa += a[i] * a[i]; sb += b[i] * b[i]; }' ' return sa === 0 || sb === 0 ? 0 : dot / Math.sqrt(sa * sb);' '}' 'const store = chunks.map(function (c) { return { id: c.id, vector: vectorOf(c.text) }; });' 'const MINIMUM = 0.3;' >> score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the scoring script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function retrieve(question) {' ' const query = vectorOf(question);' ' let best = { id: \"\", score: 0 };' ' store.forEach(function (item) {' ' const score = cosine(query, item.vector);' ' if (score > best.score) best = { id: item.id, score: score };' ' });' ' return best.score >= MINIMUM ? [best.id] : [];' '}' >> score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the scoring script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const cases = [' ' { question: \"when does the library open on saturday\", expected: \"hours-weekend\" },' ' { question: \"how many books can I borrow\", expected: \"borrowing\" },' ' { question: \"how much is a late fine\", expected: \"fines\" },' ' { question: \"can I book a study room\", expected: \"rooms\" },' ' { question: \"who is the librarian\", expected: null },' '];' 'let hits = 0;' 'cases.forEach(function (one) {' ' const found = retrieve(one.question);' ' const hit = one.expected === null ? found.length === 0 : found.indexOf(one.expected) >= 0;' ' if (hit) hits = hits + 1;' ' console.log((hit ? \"HIT \" : \"MISS \") + one.question + \" -> \" + (found.join(\" \") || \"nothing\"));' '});' 'console.log(\"Hit rate:\", hits, \"of\", cases.length);' >> score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it and read the hit rate
docker exec -w /home/node/project agent node score.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What it prints
| Line | What it shows |
|---|---|
| HIT when does the library open on saturday -> hours-weekend | The right chunk |
| HIT who is the librarian -> nothing | Nothing found, which is right |
| Hit rate: 5 of 5 | Every case was a hit |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think retrieval works if it always returns something. For a question the documents cannot answer, returning something is a miss. Count those questions too, and give the retriever a way to say nothing.