Perfect on Friday, wrong on Monday
An IT team builds a model to spot urgent support tickets. They check it on the same 500 tickets it learned from. It gets 499 right.
On Monday it goes live. It misses a server that is down and flags a request for a new chair as urgent.
The check on Friday told them nothing useful. This lesson is about why.
By the end of this lesson
Note
I can explain why we test on data the model has not seen.
Key words
| Word | What it means | Example |
|---|---|---|
| Training set | the examples a model learns from | |
| Test set | examples kept back to check the model | |
| Unseen data | examples the model never learned from | |
| Split | dividing the examples into two sets | 80% and 20% |
| Leakage | test examples getting into training |
The cells students fill in are left blank in this preview.
A model must work on new cases
A model is only useful on cases it has not met yet: tomorrow’s parcels, Monday’s tickets.
Checking it on the examples it learned from is like checking a new colleague with the exact cases from their training. They may have remembered the answers.
So you keep some examples back. The model never sees them while it learns. They show how it will do on unseen data.
The same model, checked two ways
| Checked on | Right | What it tells you |
|---|---|---|
| The 500 training tickets | 499 of 500 | Very little: it has seen them |
| 100 tickets kept back | 62 of 100 | Roughly how it will do on Monday |
The cells students fill in are left blank in this preview.
Split the examples before you train
Before training, you split the examples into two sets.
The training set is the bigger part, often 80%. The model learns from it.
The test set is the rest, often 20%. You use it once, at the end, to measure the model.
With 1,000 tickets, that is 800 to learn from and 200 to test on.
Working out a split
| Examples | Training share | Training set | Test set |
|---|---|---|---|
| 1,000 | 80% | 800 | 200 |
| 500 | 80% | 400 | 100 |
| 10 | 70% | 7 | 3 |
The cells students fill in are left blank in this preview.
Split some tickets
# 10 past support tickets, oldest first.
tickets = ["printer", "password", "server down", "new mouse", "data lost",
"email slow", "site offline", "new chair", "VPN broken", "font size"]
percent = 80
cut = len(tickets) * percent // 100
training = tickets[:cut]
testing = tickets[cut:]
print("Train on", len(training), "tickets:", training)
print("Test on", len(testing), "tickets:", testing)
# Change percent to 70, then press Run.
# How many tickets does the test set get now?
Leakage spoils the test
The test only works if the test examples stay out of training. When one gets in, that is leakage.
Leakage is easy to cause by accident. You might copy a ticket twice, or train again on all the data after testing.
A model with leakage looks better than it is. A hiring manager will expect you to check for it.
A common mistake
Warning
Many people think a high score on the training data proves a model works. It proves only that the model fits what it has seen. Trust a score only when it comes from a test set the model never learned from.