Association rules: confidence and lift

An association rule is an if-then pattern found in records, such as "if a basket holds bread, it also holds butter". Shops, libraries and websites mine rules like this from their sales and loans.

A rule is only useful if you can rate it. You will learn two ratings, confidence and lift, and how to tell a real link from a coincidence.

A rule has two sides. Written bread → butter, bread is the antecedent and butter is the consequent.

Confidence answers one question: of the baskets that hold the antecedent, what share also hold the consequent? Count the baskets with both, then divide by the baskets with the antecedent alone.

baskets = [
    ["bread", "milk", "eggs"], ["bread", "butter"], ["milk", "eggs", "tea"],
    ["bread", "milk", "butter"], ["bread", "milk", "eggs", "butter"],
    ["milk", "tea"], ["bread", "eggs"], ["bread", "milk", "butter"],
]


def count(items):
    total = 0
    for basket in baskets:
        found = True
        for item in items:
            if item not in basket:
                found = False
        if found:
            total = total + 1
    return total


def confidence(first, then):
    return count([first, then]) / count([first])


print("bread then butter:", round(confidence("bread", "butter"), 3))
print("butter then bread:", round(confidence("butter", "bread"), 3))
# Add a line for milk then eggs. Work out the answer by hand first.

Confidence can be high only because the consequent is common. If 75 baskets in 100 hold milk, then "anything → milk" is right 75 times in 100 by luck.

Lift removes the luck: confidence divided by the support of the consequent. A lift of 1 means no link. Above 1, the items go together more than chance. Below 1, less.

docker exec jupyter python3 -c "print(round(4 / 6, 3), round((4 / 6) / (6 / 8), 3))"

The lab's bike-hire days make a good test. Take the rule "sunny → busy", where busy means 300 hires or more.

The next command counts the sunny days that were busy. It divides by the sunny days for the confidence. Then it divides by the share of busy days for the lift.

docker exec jupyter python3 -c "import csv; days = list(csv.DictReader(open('hires.csv'))); sunny = [d for d in days if d['weather'] == 'sunny']; busy = [d for d in days if int(d['hires']) >= 300]; both = [d for d in sunny if int(d['hires']) >= 300]; conf = len(both) / len(sunny); print(round(conf, 3), round(conf / (len(busy) / len(days)), 3))"

A rule seen in only one basket can have a huge lift and mean nothing. Find such a rule in the stationery baskets of the assignment, then explain why a minimum support is worked out before any rule is rated.

Then try the rule weekend → busy on the lab data and compare its lift with sunny → busy.