A model right 80% of the time
Judging a model means measuring how often its predictions are right, and which kinds of mistake it makes. A confusion matrix, precision and recall do this better than accuracy alone.
A library predicts which loans will come back late, so that it can send a reminder. A model that never predicts a late loan can still score 80%, and it is useless. You will learn to see why.
By the end of this lesson
Note
I can judge a model with a confusion matrix, precision and recall on a test set.
Key words
| Word | What it means |
|---|---|
| Confusion matrix | a table counting right and wrong predictions by kind |
| Precision | the share of positive predictions that were right |
| Recall | the share of real positives the model found |
| Test set | rows kept back to judge the model |
| Accuracy | the share of all predictions that were right |
The cells students fill in are left blank in this preview.
A confusion matrix counts four kinds
Pick one outcome to be the positive class. Here it is "returned late". Every prediction then falls in one of four cells.
A true positive is a late loan the model flagged. A false positive is an on-time loan it flagged. A false negative is a late loan it missed. A true negative is an on-time loan it left alone.
Worked example: twenty loans kept back
| Flagged late | Not flagged | Total | |
|---|---|---|---|
| Really late | 3 (true positives) | 1 (false negative) | 4 |
| Really on time | 2 (false positives) | 14 (true negatives) | 16 |
| Total | 5 | 15 | 20 |
The cells students fill in are left blank in this preview.
Count the four cells with Python
docker exec jupyter python3 -c "a = 'ooLoooooLoooooooLooL'; p = 'ooLooLooLoooLooooooL'; print(sum(x == 'L' and y == 'L' for x, y in zip(a, p)), sum(x == 'o' and y == 'L' for x, y in zip(a, p)), sum(x == 'L' and y == 'o' for x, y in zip(a, p)), sum(x == 'o' and y == 'o' for x, y in zip(a, p)))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Precision and recall ask different questions
Precision asks: when the model says late, how often is it right? It is true positives divided by everything flagged.
Recall asks: of the loans that really were late, how many did the model find? It is true positives divided by everything really late. Low precision wastes reminders. Low recall lets late loans slip by.
Work out the three figures
tp = 3 # flagged late, and it was late
fp = 2 # flagged late, but it was on time
fn = 1 # not flagged, but it was late
tn = 14 # not flagged, and it was on time
total = tp + fp + fn + tn
print("accuracy:", (tp + tn) / total)
print("precision:", tp / (tp + fp))
print("recall:", tp / (tp + fn))
print("never saying late:", (fp + tn) / total)
# Change fn to 3 and tn to 12. Which figure falls, and which stays the same?
Check precision and recall
docker exec jupyter python3 -c "print(round(3 / 5, 2), round(3 / 4, 2))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think precision and recall are two names for the same score. They divide by different counts. Precision looks at what the model flagged. Recall looks at what really happened. A model can be high in one and low in the other.
A test set holds rows kept back
A model can memorise the rows it learns from. So we keep some rows back, the test set, and score the model on those alone.
The figure from the learning rows tells you how well the model remembers. The figure from the test set tells you how well it will do on new loans. Quote the second one.
Keep every fourth loan back
loans = list(range(1, 21))
train = []
test = []
for number in loans:
if number % 4 == 0:
test.append(number)
else:
train.append(number)
print("train:", len(train), "test:", len(test))
print("test loans:", test)
# Change 4 to 5. How many loans are kept back now?
In your notebook: judge a tree
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
hires = pd.read_csv("hires.csv")
hires["weekend"] = hires["weekday"].isin(["Sat", "Sun"]).astype(int)
X = hires[["temp_c", "rain_mm", "weekend"]]
y = hires["hires"] >= 300
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
print(classification_report(y_test, tree.predict(X_test)))
Going further
Read the notebook's report. Which class has the lower recall, and what would that mean for a hire shop?
Then explain which you would prefer for a model that spots bank fraud: high precision, high recall or both, and what each mistake costs.