Which words matter most in a document?
TF-IDF, short for term frequency times inverse document frequency, is a score for how special a word is in a document. A word that is common in one document but rare across the collection scores high.
Search systems use it to put the best match first. By the end you will work the score out by hand and in Python.
By the end of this lesson
Note
I can rank documents by how special a word is in them.
Key words
| Word | What it means | Example |
|---|---|---|
| Term | a word that is searched for or counted | |
| Term frequency | how often a word appears in one document, as a share of its words | 1 of 4 is 0.25 |
| Document frequency | how many documents contain the word | |
| Inverse document frequency | a number that is high for rare words and 0 for words in every document | |
| Score | one number that says how well a document matches a query |
The cells students fill in are left blank in this preview.
Why is counting matches not enough?
Suppose you search for "the bus is late". Almost every document contains "the" and "is", so those words match everything and tell you nothing.
A good score gives a word weight by how well it picks out documents. A word in every document gets none. A word in only one gets the most.
What is term frequency?
Term frequency is how often a word appears in a document, divided by the number of words in that document. Dividing stops a long document winning just for being long.
In "bus is late today" the word "bus" appears once in four words. Its term frequency is 1 divided by 4, which is 0.25.
What is inverse document frequency?
Count the documents that contain the word. This is its document frequency. Then work out the log, to base 10, of the number of documents divided by that count.
That is the inverse document frequency. The log stops a very rare word from dominating. A word in every document gives log of 1, which is 0.
Work out three idf values
docker exec jupyter python3 -c "import math; print(round(math.log10(4/2),3), round(math.log10(4/1),3), round(math.log10(4/4),3))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: four documents and their idf
| Word | In how many of 4 documents | 4 divided by that | idf (log base 10) |
|---|---|---|---|
| is | 4 | 1 | 0 |
| today | 3 | 1.33 | 0.125 |
| bus | 2 | 2 | 0.301 |
| late | 2 | 2 | 0.301 |
| train | 1 | 4 | 0.602 |
| library | 1 | 4 | 0.602 |
The cells students fill in are left blank in this preview.
Count document frequency in Python
import math
docs = [
["bus", "is", "late", "today"],
["train", "is", "late", "today"],
["bus", "timetable", "is", "new"],
["the", "library", "is", "open", "today"],
]
for word in ["is", "today", "bus", "train"]:
df = 0
for doc in docs:
if word in doc:
df = df + 1
print(word, df, round(math.log10(len(docs) / df), 3))
# Add "the" to the list of words. How rare is it?
How do you put the two parts together?
Multiply the term frequency by the inverse document frequency. That is the tf-idf of a word in a document.
It is high when the word is frequent in this document and rare elsewhere. It is 0 when the word is missing, or when it is in every document.
Worked example: scoring the query "late bus"
| Document | tf-idf of late | tf-idf of bus | Score |
|---|---|---|---|
| 1: bus is late today | 0.25 x 0.301 = 0.075 | 0.25 x 0.301 = 0.075 | 0.151 (the parts add to 0.1505) |
| 2: train is late today | 0.25 x 0.301 = 0.075 | 0 | 0.075 |
| 3: bus timetable is new | 0 | 0.25 x 0.301 = 0.075 | 0.075 |
| 4: the library is open today | 0 | 0 | 0 |
The cells students fill in are left blank in this preview.
Work out the tf-idf of one word
docker exec jupyter python3 -c "import math; print(round(0.25 * math.log10(4/1), 3))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Score every document for a query
import math
docs = [
["bus", "is", "late", "today"],
["train", "is", "late", "today"],
["bus", "timetable", "is", "new"],
["the", "library", "is", "open", "today"],
]
def tf_idf(word, doc):
df = 0
for d in docs:
if word in d:
df = df + 1
if df == 0:
return 0
return doc.count(word) / len(doc) * math.log10(len(docs) / df)
for n in range(len(docs)):
score = tf_idf("late", docs[n]) + tf_idf("bus", docs[n])
print(n + 1, round(score, 3))
# Search for "train library" instead. Which document wins?
A common mistake
Warning
Many people think the most frequent word is the most important. In "the bus is late", the words "the" and "is" may be the most frequent, yet their tf-idf is 0 if every document has them. A word matters when it is frequent here and rare elsewhere.
What is tf-idf used for?
A search system adds up the tf-idf of each query word in each document and shows the highest scores first. A mail filter can use it to pick out the words that make one message different from the rest.
The exact formula varies. The order of the results is what matters, and that stays close from one variant to the next.
Going further
Add a fifth document to the collection, "the bus is late again". Work out what happens to the idf of "bus" and of "is" by hand, then run the code to check.
Then explain why a very long document could win a search without term frequency being divided by its length.