Retrieval metrics: recall and reciprocal rank
Retrieval metrics score the search part of an AI feature on its own. Many features search a set of documents and then answer from what they find. When an answer is wrong, these scores show which part failed: the search or the answer.
In this lesson the search is a simple keyword counter written in code.
By the end of this lesson
Note
I can score the search apart from the answer.
Key words
| Word | What it means | Example |
|---|---|---|
| Retriever | the part that finds documents for a question | |
| Ranked list | documents in order, best first | |
| Relevant | a document that really holds the answer | |
| Recall at k | the share of relevant documents in the top k | 0.5 |
| Reciprocal rank | one divided by the rank of the first relevant document | 0.5 |
The cells students fill in are left blank in this preview.
A keyword retriever counts shared words
Take the question and drop small words such as the and is. For each document, count how many question words it holds. Sort by that count, best first.
It is simple, and it has limits. It does not know that opens and open are one word, or that extend means renew.
Worked example: one question, three documents
| Question words | Document | Words shared | Rank |
|---|---|---|---|
| cost, borrow, books, late | Late books cost twenty pence a day | 3 | 1 |
| cost, borrow, books, late | Members may borrow six books | 2 | 2 |
| cost, borrow, books, late | Ask the desk to renew books | 1 | 3 |
The cells students fill in are left blank in this preview.
Recall and rank score the list
Recall at k asks how many of the right documents are in the top k. Two are right and one is in the top 2: recall at 2 is 0.5.
Reciprocal rank looks at the first right document only. At rank 1 it scores 1. At rank 2 it scores 0.5. At rank 4 it scores 0.25. The mean over many questions is the mean reciprocal rank.
Worked example: five questions
| Question | Right document at rank | Reciprocal rank | Recall at 1 |
|---|---|---|---|
| library opens | 1 | 1 | 1 |
| library closed (two right) | 1 and 2 | 1 | 0.5 |
| how long can I keep books before they are late | 2 | 0.5 | 0 |
| extend my loan | not found | 0 | 0 |
| print a page | 1 | 1 | 1 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a wrong answer means the model is poor. If the right document was never found, a perfect model still cannot answer. Score the search first, then the answer.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the keyword search, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const docs = {' ' d1: \"The library opens at nine and closes at five\",' ' d2: \"Members may borrow six books for three weeks\",' ' d3: \"Late books cost twenty pence a day\",' ' d4: \"The cafe sells soup and tea\",' ' d5: \"Computers are free to use for one hour\",' ' d6: \"Ask the desk to renew books online\",' ' d7: \"The library is closed on Sundays\",' ' d8: \"Printing costs ten pence a page\",' '};' > search.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the keyword search, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const stop = [\"the\", \"a\", \"an\", \"at\", \"and\", \"to\", \"for\", \"of\", \"is\", \"are\", \"how\", \"when\", \"can\", \"i\", \"my\", \"on\", \"does\"];' 'function tokens(text) {' ' return text.toLowerCase().split(\" \").filter(function (w) { return w.length > 0 && stop.indexOf(w) < 0; });' '}' 'function search(query) {' ' const q = tokens(query);' ' const scored = Object.keys(docs).map(function (id) {' ' const have = tokens(docs[id]);' ' return { id: id, score: q.filter(function (w) { return have.indexOf(w) >= 0; }).length };' ' }).filter(function (s) { return s.score > 0; });' ' scored.sort(function (a, b) { return b.score - a.score || (a.id < b.id ? -1 : 1); });' ' return scored.map(function (s) { return s.id; });' '}' >> search.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the keyword search, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const queries = [' ' [\"library opens\", [\"d1\"]], [\"library closed\", [\"d1\", \"d7\"]], [\"how long can I keep books before they are late\", [\"d2\"]],' ' [\"extend my loan\", [\"d6\"]], [\"print a page\", [\"d8\"]],' '];' 'let sumRr = 0;' 'let sumRecall = 0;' 'queries.forEach(function (item) {' ' const ranked = search(item[0]);' ' const at = ranked.findIndex(function (id) { return item[1].indexOf(id) >= 0; });' ' const rr = at < 0 ? 0 : 1 / (at + 1);' ' const hit = ranked.slice(0, 1).filter(function (id) { return item[1].indexOf(id) >= 0; }).length;' ' sumRr += rr;' ' sumRecall += hit / item[1].length;' ' console.log(item[0] + \" -> \" + (ranked.join(\",\") || \"nothing\") + \" rr \" + rr.toFixed(2));' '});' >> search.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the keyword search, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"mean reciprocal rank: \" + (sumRr / queries.length).toFixed(2));' ' console.log(\"mean recall at 1: \" + (sumRecall / queries.length).toFixed(2));' ' const q = \"Using only this text, answer in one sentence: \" + docs.d3 + \". Question: how long can I keep books before they are late?\";' ' const sample = \"The text does not say. It only says late books cost twenty pence a day.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"answer from the top document: \" + reply.text);' '}' 'main();' >> search.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the search and score it
docker exec -w /home/node/project agent node search.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| library opens -> d1,d7 rr 1.00 | The ranked list and its rank score |
| how long can I keep books before they are late -> d3,d2,d6 rr 0.50 | The right document is second |
| extend my loan -> nothing rr 0.00 | A synonym the counter cannot see |
| mean reciprocal rank: 0.70 | The mean over five questions |
| answer from the top document: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.