Spotting a model that learned by heart

An estate agent’s team trains a model to guess house prices. On the houses it learned from, it is never wrong.

On new houses it is wrong about half the time. One flat is priced at twice its worth.

The model did not learn what makes a house cost more. It learned the answers by heart.

A good model learns the pattern: bigger houses near the station cost more.

An overfitting model learns every detail of its examples, even the noise. Number 12 sold for more because the buyer was in a hurry. That is chance, not a pattern.

The model then does very well on the examples and badly on anything new.

# A "model" that only remembers: it looks each house up.
training = {"12 Mill Lane": 210000, "4 High Street": 185000, "9 Park Road": 320000}

def predict_price(address):
    if address in training:
        return training[address]
    return 0  # It has never seen this house, so it has no idea.

print("Training house:", predict_price("4 High Street"))
print("Training house:", predict_price("9 Park Road"))
print("New house:", predict_price("7 Station Road"))

# Add "7 Station Road" with a price to training, then press Run.
# Did the model get better at houses it has never seen?

You spot overfitting by comparing two scores from the last two lessons.

Measure the model on the training set, then on the test set. Subtract the test score from the training score. That is the gap.

A small gap is normal. A big gap means the model remembered its examples. If both scores are low, the model is underfitting: it has not learned enough.

For overfitting, give the model more examples, so chance details matter less. Or use a simpler model that cannot remember so much. Or stop training sooner.

For underfitting, do the opposite. Give the model more useful features, or a model that can learn more.

Then measure both scores again.