A score that was too good
Data leakage is when information from the test rows reaches a model before it is tested, so its score looks better than it really is. It matters because a leaked score promises more than the model can deliver.
A team tests a model on rows kept back from training and is told it is right nine times in ten. In use it is right seven times in ten. The rows it was tested on were not really new.
By the end of this lesson
Note
I can keep a test set clean and spot when information has leaked into it.
Key words
| Word | What it means |
|---|---|
| Test set | rows kept back and looked at once, at the very end |
| Validation set | rows used to choose between models and settings |
| Leakage | information from the test rows, or from the answer, reaching the model before the test |
| Duplicate | a row that appears more than once |
| Baseline | the score of a very simple guess, to compare with |
The cells students fill in are left blank in this preview.
Three sets, three jobs
Last lesson you kept a validation set back. You looked at it many times, to decide when to stop and which settings to use.
That makes it a little less fresh every time. So a careful team keeps a third set, the test set. It is used once, at the end, for the score you report.
Worked example: a year in three sets
| Set | Share | Days | Used for |
|---|---|---|---|
| Training set | 60 percent | 219 | Setting the weights |
| Validation set | 20 percent | 73 | Choosing a model and its settings |
| Test set | 20 percent | 73 | The one honest score at the end |
The cells students fill in are left blank in this preview.
Work out the three sizes
docker exec jupyter python3 -c "print(int(365 * 0.6), int(365 * 0.2), 365 - int(365 * 0.6) - int(365 * 0.2))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Leakage: the test is not really new
Leakage is any way that the model gets to see test information before the test. The easiest one is a duplicate: the same row in the training set and in the test set.
A model that only remembers rows then looks excellent. Run the code and see it happen with a model that does nothing but remember.
A model that only remembers
# Eight days: (temperature, rain) and whether the day was busy.
days = [
((3, 0), "quiet"), ((8, 0), "busy"), ((12, 2), "quiet"), ((15, 0), "busy"),
((18, 0), "busy"), ((9, 5), "quiet"), ((21, 0), "busy"), ((6, 3), "quiet"),
]
train = days[:6]
fresh = days[6:]
copies = days[:2] + days[6:]
def memoriser(day):
# Remembers every training day exactly. A day it has not seen gets "quiet".
for known, label in train:
if known == day:
return label
return "quiet"
def score(rows):
right = 0
for day, label in rows:
if memoriser(day) == label:
right = right + 1
return right / len(rows)
print("on its own training days:", score(train))
print("on two new days:", score(fresh))
print("on two copies and two new days:", score(copies))
# Which score would you believe? Change one new day and run again.
The same test on a real table
Now the same idea on the lab's bike hires. A 1-nearest-neighbour model answers with the label of the closest day it knows, so it is close to a memoriser. A busy day is one with 300 hires or more.
First write a small script into the lab. It takes one word: honest, copies or answer.
Write the script into the lab
docker exec jupyter sh -c "printf '%s\n' 'import sys' 'import pandas as pd' 'from sklearn.model_selection import train_test_split' 'from sklearn.neighbors import KNeighborsClassifier' '' 'hires = pd.read_csv(\"hires.csv\")' 'way = sys.argv[1]' 'if way == \"copies\":' ' hires = pd.concat([hires, hires], ignore_index=True)' 'columns = [\"temp_c\", \"rain_mm\"]' 'if way == \"answer\":' ' columns = columns + [\"hires\"]' 'label = hires[\"hires\"] >= 300' 'X_train, X_test, y_train, y_test = train_test_split(hires[columns], label, test_size=0.25, random_state=1)' 'model = KNeighborsClassifier(n_neighbors=1).fit(X_train, y_train)' 'print(way, \"learnt from:\", round(model.score(X_train, y_train), 3), \"kept back:\", round(model.score(X_test, y_test), 3))' > steps/leak.py"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it honestly
docker exec jupyter python3 steps/leak.py honestShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write every day twice, then split
docker exec jupyter python3 steps/leak.py copiesShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A column that gives the answer away
The other common leak is a column that holds the answer, or something known only afterwards. A refund flag in a table that predicts refunds is one. So is a total that is added up after the event.
In the lab, hires is the number the busy label is made from. Put it among the inputs and see.
Give the model the hires column
docker exec jupyter python3 steps/leak.py answerShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a high score means a good model. It may only mean the test was easy. A perfect score on kept-back rows is a reason to look for a leak, not to celebrate. Compare every score with a baseline, such as always guessing quiet.
Worked example: three leaks and checks
| What happened | Why it leaks | How to check |
|---|---|---|
| Every day written twice, then split | A test row has a twin in the training set | Count test rows that also appear in the training set |
| The hires column used to predict a label made from hires | The input holds the answer | Ask of each column: is it known before the answer? |
| Ten models tried, the best test score reported | The test set was used to choose | Choose with the validation set, test once |
The cells students fill in are left blank in this preview.
Going further: leakage in your own notebook
Take the lab's hires.csv and a model of your own. Write the score on the days it learnt from and on the days kept back.
Then add one column that you think leaks, and run it again. How far does the score move? Write one sentence saying why that column would not exist on the day you really needed a prediction.