Gradient descent and the learning rate

Gradient descent is how a neural network learns from its errors. Each small step makes the network a little less wrong, and thousands of steps are what training means.

The network measures how wrong it is, works out which way to change each weight, and steps that way. You can follow every step with one weight and two days of bike hires.

Our toy model has one weight. Hires in hundreds equal the weight times the temperature in tens of degrees. Two days are known: 10 degrees gave 2 hundred hires, and 20 degrees gave 4.

For each day, the error is the guess minus the truth. Square it, then take the mean. That mean is the loss. Squaring stops errors cancelling out, and big errors count for more.

# One weight, two days: temperature in tens of degrees and hires in hundreds.
xs = [1, 2]
ys = [2, 4]
w = 0

total = 0
for i in range(2):
    error = w * xs[i] - ys[i]
    total = total + error * error
print("loss:", total / 2)
# Change w to 1.5 and press Run. Is the loss lower?

Think of the loss as a valley, with the weight as your position. The gradient is the slope where you stand. Its sign says which way the loss rises. Its size says how steep it is.

For this model, work out 2 times the error times the input for each day, then take the mean. At a weight of 0 the gradient is -10: the loss falls steeply as the weight goes up.

docker exec jupyter python3 -c "xs = [1, 2]; ys = [2, 4]; w = 1; print(sum(2 * (w * x - y) * x for x, y in zip(xs, ys)) / 2)"

One step makes the new weight the old weight minus the learning rate times the gradient. The minus sign sends the weight downhill.

A small learning rate takes tiny steps and is slow. A huge one jumps over the lowest point and can make the loss worse. Repeating the step is gradient descent.

xs = [1, 2]
ys = [2, 4]
rate = 0.1
w = 0

for step in range(4):
    gradient = 0
    for i in range(2):
        error = w * xs[i] - ys[i]
        gradient = gradient + 2 * error * xs[i]
    gradient = gradient / 2
    w = w - rate * gradient
    print("step", step + 1, "weight", round(w, 4))
# Change the rate to 0.5 and press Run.
# Then try 0.2. What happens at the first step?

docker exec jupyter python3 -c "w = 1.5; g = sum(2 * (w * x - y) * x for x, y in zip([1, 2], [2, 4])) / 2; print(g, round(w - 0.1 * g, 4))"

A real network has thousands of weights, and each needs its own gradient. Backpropagation finds them all in one sweep.

The forward pass gives the loss. Then the sweep goes backwards from the output layer. Each layer passes back how much it added to the error. One pass forward and one back give every gradient.

In the descent code, set the rate to 0.25 and run ten steps instead of four. Does the weight settle at 2?

Then use days where the best weight is 3, with ys = [3, 6]. Work out the first gradient by hand before you run it.