How do you judge a search?
Precision and recall are two numbers that judge a search. Precision asks how many of the results were right. Recall asks how many of the right documents were found.
You need both, because a search can score well on one and badly on the other. By the end you will work both out and explain the trade-off.
By the end of this lesson
Note
I can judge a search with precision and recall.
Key words
| Word | What it means | Example |
|---|---|---|
| Relevant | a document a person judges to answer the query | |
| Retrieved | a document the search returned | |
| Precision | the share of the retrieved documents that are relevant | |
| Recall | the share of the relevant documents that were retrieved | |
| Precision at k | precision counted over only the first k results | top 3 |
The cells students fill in are left blank in this preview.
Why does a search need to be judged?
A ranking can look sensible and still be poor. To judge it fairly, a person first decides which documents are relevant to a query. That list is the answer key.
Then you compare what the search retrieved with that key. Two questions come out of the comparison, and each has its own number.
What are precision and recall?
Precision is the number of relevant documents retrieved, divided by the number retrieved. It asks: of what you showed, how much was right?
Recall is the number of relevant documents retrieved, divided by the number of relevant documents. It asks: of what was right, how much did you show?
Worked example: one search judged
| Item | Documents | How many |
|---|---|---|
| Retrieved, in rank order | 3, 1, 7, 2, 9 | 5 |
| Relevant, chosen by a person | 1, 2, 3, 5 | 4 |
| Retrieved and relevant | 3, 1, 2 | 3 |
| Precision | 3 divided by 5 | 0.6 |
| Recall | 3 divided by 4 | 0.75 |
The cells students fill in are left blank in this preview.
Work out precision and recall
docker exec jupyter python3 -c "print(3 / 5, 3 / 4)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Count the hits in Python
retrieved = [3, 1, 7, 2, 9]
relevant = [1, 2, 3, 5]
hits = 0
for doc_id in retrieved:
if doc_id in relevant:
hits = hits + 1
print("hits", hits)
print("precision", hits / len(retrieved))
print("recall", hits / len(relevant))
# Remove 9 from retrieved. What happens to precision and to recall?
Why do precision and recall pull apart?
Show everything and recall is 1, because every relevant document is in the list. Precision falls, because most of the list is wrong. Show one safe result and precision may be 1 while recall is tiny.
Pushing one up usually pushes the other down. A good ranking puts the relevant documents first.
What is precision at k?
People look at the top of a list. Precision at k counts only the first k results. It is the number of relevant documents in the top k, divided by k.
It suits a search page of ten results better than a figure over thousands that nobody reads.
Worked example: the same search at each k
| k | Top k results | Relevant in them | Precision at k | Recall at k |
|---|---|---|---|---|
| 1 | 3 | 1 | 1 | 0.25 |
| 2 | 3, 1 | 2 | 1 | 0.5 |
| 3 | 3, 1, 7 | 2 | 0.667 | 0.5 |
| 4 | 3, 1, 7, 2 | 3 | 0.75 | 0.75 |
| 5 | 3, 1, 7, 2, 9 | 3 | 0.6 | 0.75 |
The cells students fill in are left blank in this preview.
Work out precision at k for each k
docker exec jupyter python3 -c "r=[3,1,7,2,9]; rel=[1,2,3,5]; print([round(len([x for x in r[:k] if x in rel])/k,3) for k in range(1,6)])"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Precision and recall at each k
retrieved = [3, 1, 7, 2, 9]
relevant = [1, 2, 3, 5]
for k in range(1, 6):
hits = 0
for doc_id in retrieved[:k]:
if doc_id in relevant:
hits = hits + 1
print(k, round(hits / k, 3), round(hits / len(relevant), 3))
# The columns are k, precision at k and recall at k. Add 5 at the front of the list.
A common mistake
Warning
Many people think a high precision means a good search. Return one safe document and precision is 1, yet in our example recall is only 0.25. Always give both numbers, or one that combines them such as the F score.
Can one number combine them?
The F score is 2 times precision times recall, divided by precision plus recall. It is high only when both are high.
Our search has precision 0.6 and recall 0.75, so the F score is 0.9 divided by 1.35, which is 0.667. Someone still has to judge the documents first: the numbers are only as good as that judgement.
Going further
Choose a query for your own college library, such as "revision guides". List ten results and decide for each if it is relevant. Work out precision at 5 and precision at 10.
Then explain why two people may judge the same results differently, and what that does to the numbers.