Classifying text with word counts

Text classification means giving a text a label, such as spam or work, from the words it contains. A naive Bayes classifier does it with nothing more than word counts and a little probability.

It is called naive because it treats every word as if it were unrelated to the others. That is not true, and it still works well enough to filter email.

You need a training set: texts that people have already labelled. The program counts how often each word appears under each label.

Words such as "free", "prize" and "click" will be common under spam. Words such as "meeting" and "report" will be common under work.

messages = ["win a free prize now", "free voucher click now", "meeting moved to monday"]
labels = ["spam", "spam", "work"]
counts = {"spam": {}, "work": {}}
for n in range(len(messages)):
    for word in messages[n].split():
        table = counts[labels[n]]
        if word in table:
            table[word] = table[word] + 1
        else:
            table[word] = 1
print(counts["spam"])
# Add "free lunch at noon" as work. Does "free" appear under work now?

docker exec jupyter sh -c "printf 'win a free prize now\nfree voucher click now\nmeeting moved to monday\n' | grep -c free"

The chance of a word under a label is its count there, divided by all the tokens under that label. A word never seen under a label would get 0, and one zero would wipe out the whole answer.

Smoothing fixes that: add 1 to every count, and add the size of the vocabulary to the total.

import math

spam_free = 3        # "free" appears 3 times in the spam messages
spam_total = 13      # the spam messages hold 13 tokens in all
vocabulary = 27      # 27 different words in all the messages
chance = (spam_free + 1) / (spam_total + vocabulary)
print(round(chance, 3))
print(round(math.log(chance), 3))
# Use 0 for spam_free. Is the chance still above 0?

To score a text under a label, start with the prior: the share of training texts with that label. Then multiply by the chance of each word. Many small chances multiplied make a tiny number.

So take the logarithm of each chance and add them. That is the log score. The label with the highest score wins.

docker exec jupyter python3 -c "import math; print(round(math.log(3 / 7), 3), round(math.log(4 / 40), 3))"

It ignores order, so "not free" looks like "free". It only knows the words in its training set, so a new scam wording may slip through.

Judge it on messages it has not seen, and compare it with a baseline. A filter that always says "work" is right most of the time if most mail is work.