Overfitting and early stopping

Overfitting happens when a neural network learns its training examples so closely that it does badly on new ones. Spotting it, and stopping training at the right moment, separates a model that works in a demo from one that works in a shop.

A network with enough weights can memorise a whole year of bike hires, the random noise included.

A network is judged on rows it has never seen. So you split the data first. The training set sets the weights. The validation set is kept back, and you only use it to measure the loss.

A common split is 80 percent for training and 20 percent for validation. A year of 365 days gives 292 and 73.

docker exec jupyter wc -l hires.csv

docker exec jupyter python3 -c "print(int(365 * 0.8), 365 - int(365 * 0.8))"

Early in training both losses fall. Then the network starts to fit the noise in the training rows. The training loss keeps falling, but the validation loss stops falling and starts to rise.

The distance between the two is the gap. A growing gap is the sign of overfitting.

train = [9.0, 6.0, 4.0, 2.5, 1.5, 0.9, 0.5, 0.2]
validation = [9.5, 6.5, 4.8, 4.0, 3.8, 4.0, 4.5, 5.2]

best = validation[0]
best_epoch = 1
for n in range(8):
    gap = validation[n] - train[n]
    print("epoch", n + 1, "gap", round(gap, 1))
    if validation[n] < best:
        best = validation[n]
        best_epoch = n + 1
print("best epoch:", best_epoch)
# Change the last validation loss from 5.2 to 3.0 and press Run.
# Which epoch is best now?

Early stopping ends training once the validation loss has stopped improving. Patience is how many epochs in a row you wait without a new best before you stop.

Then you keep the weights from the best epoch, not the last one. The last ones have already started to overfit.

Early stopping is one fix. Dropout switches off a random share of neurons at each training step, so none can lean on another. A weight penalty adds a cost to the loss for large weights.

You can also add more data, or use a smaller network that has fewer weights to memorise with.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler

hires = pd.read_csv("hires.csv")
hires["weekend"] = hires["weekday"].isin(["Sat", "Sun"]).astype(int)
label = (hires["hires"] >= 300).astype(int)
X = StandardScaler().fit_transform(hires[["temp_c", "rain_mm", "weekend"]])
X_train, X_val, y_train, y_val = train_test_split(X, label, test_size=0.2, random_state=1)

for size in [(2,), (64, 64)]:
    net = MLPClassifier(hidden_layer_sizes=size, max_iter=3000, random_state=0)
    net.fit(X_train, y_train)
    print(size, "training:", round(net.score(X_train, y_train), 3),
          "validation:", round(net.score(X_val, y_val), 3))
# Then add early_stopping=True to MLPClassifier and run it again.

A final test set is a third group of rows that you use once, at the very end. Why can you not use the validation set for that job?

Hint: you chose the stopping epoch by looking at the validation loss, so the validation set has already helped to pick the model.