How a small neural network learns

A booking app tells the shop that Saturday will be busy. Nobody at the app wrote "if it is warm and dry at the weekend". The app started with numbers that meant nothing and corrected them, one past day at a time.

That correcting is what "learning" means for a neural network. It is arithmetic you can follow by hand.

A perceptron is one neuron. It multiplies each input by its weight, adds them up, and adds the bias.

If the total is above 0, it answers 1, here "busy". Otherwise it answers 0, "quiet". A big positive weight means that input pushes toward busy. A negative weight pushes toward quiet.

# One busy day: temperature in tens of degrees, rain in cm, weekend.
day = [1.8, 0.0, 1]
target = 1
weights = [0.0, 0.0, 0.0]
bias = 0.0
rate = 0.1

total = bias
for n in range(3):
    total = total + weights[n] * day[n]
guess = 0
if total > 0:
    guess = 1
error = target - guess
for n in range(3):
    weights[n] = round(weights[n] + rate * error * day[n], 2)
bias = round(bias + rate * error, 2)
print("guess:", guess, "error:", error)
print("weights:", weights, "bias:", bias)
# Copy the new weights and bias into lines 4 and 5.
# Run again. Is the error now 0?

Training starts with weights of 0. For each row, the perceptron guesses. The error is the true answer minus the guess: 1, 0 or -1.

Each weight then moves by the learning rate times the error times its input. A pass over every row is an epoch. Training stops after a set number of epochs, or once an epoch changes almost nothing.

docker exec jupyter python3 steps/perceptron.py

A perceptron splits its inputs with one straight boundary. Some patterns cannot be split that way.

The classic one is XOR: the answer is 1 when exactly one of two inputs is 1. No single straight line puts the two 1s on one side and the two 0s on the other. The notebook block below tries it.

from sklearn.linear_model import Perceptron
from sklearn.neural_network import MLPClassifier

X = [[0, 0], [0, 1], [1, 0], [1, 1]]
y = [0, 1, 1, 0]
print(Perceptron(random_state=0).fit(X, y).score(X, y))
# A hidden layer of 4 neurons.
net = MLPClassifier(hidden_layer_sizes=(4,), solver="lbfgs", max_iter=1000, random_state=1).fit(X, y)
print(net.score(X, y))

A hidden layer is a row of neurons between the inputs and the answer. Each draws its own boundary, and the last neuron combines them. That is how XOR is solved.

Networks use smooth decisions instead of a hard step, so each weight can be nudged a little in the direction that lowers the error. The error is passed back from the answer through each layer. This is called backpropagation.

docker exec jupyter python3 steps/network.py

In your notebook, train the perceptron from perceptron.py with three learning rates, using eta0=0.01, eta0=1 and eta0=10. Note the passes and the accuracy each time.

Then explain in two sentences why the learning rate matters more for some models than others. Hint: what happens to the decision if every weight and the bias are multiplied by the same number?