Bag of words and TF-IDF weights

A bag of words describes a text by counting how often each word appears and ignoring the order. TF-IDF (term frequency, inverse document frequency) goes one step further and gives more weight to words that are rare across texts.

With these two ideas a program can say which words a text is really about. Search engines and spam filters both start here.

Imagine tipping the words of a text into a bag and shaking it. You can still count each word, but you can no longer say which came first.

In Python a dictionary does the job. Each word is a key and its count is the value. "The bus is late" and "Late is the bus" give the same bag.

text = "the bus is late the bus is full the train is late"
counts = {}
for word in text.split():
    if word in counts:
        counts[word] = counts[word] + 1
    else:
        counts[word] = 1
print(counts)
# Change the text. Which word has the biggest count now?

docker exec jupyter python3 -c "from collections import Counter; print(Counter('the bus is late the bus is full'.split()).most_common(2))"

docker exec jupyter sh -c "cut -d, -f4 hires.csv | sort | uniq -c"

A raw count favours long texts. A word appearing 3 times in a book means less than 3 times in a short note.

Term frequency divides the count by the number of tokens: the share of the text the word takes. 2 late in 4 tokens is 0.5. 2 late in 10 tokens is 0.2.

short = "late bus".split()
long = "the bus is late and the train is late too".split()
for tokens in [short, long]:
    count = tokens.count("late")
    print(len(tokens), "tokens,", count, "late, term frequency", round(count / len(tokens), 3))
# Make the long text longer. What happens to the term frequency?

The word "the" is in nearly every text, so it cannot tell texts apart. A rare word can.

Document frequency counts the texts that contain a word. Inverse document frequency is log10 of the number of texts divided by that count. It is 0 for a word in every text, and bigger for rarer words.

docker exec jupyter python3 -c "import math; print(round(math.log10(4 / 2), 3), round(math.log10(4 / 1), 3))"

The TF-IDF weight of a word in a text is its term frequency times its inverse document frequency. A word scores high when it is common in this text and rare in the others.

The table works it out for the text "the bus is late and the train is late", from a set of four short texts.

Four texts are a tiny collection. In a real collection of thousands, "and" would be in most of them and its weight would fall towards 0.

Add three more short texts to the set in the assignment. Which weights change, and why?