How gradient descent works

Gradient descent finds the lowest point of a function by taking small steps downhill. Training a machine learning model uses it to make errors smaller.

In this lesson you will take the steps by hand and then by loop. You will also see why the size of each step, the learning rate, decides whether it works.

Stand at a point. If the slope is positive the function rises to the right, so move left. If it is negative, move right. Both are the same move: subtract the slope.

The update is new x = x - rate * slope. The learning rate is a small number such as 0.1 that stops the step from being too big.

docker exec jupyter python3 -c "exec('x = 0.0\nfor n in range(1, 6):\n    x = x - 0.1 * 2 * (x - 3)\n    print(n, round(x, 3))')"

The steps never reach the minimum exactly. They get closer each time, so you need a rule for when to stop.

Stop when the slope is smaller than a tolerance, such as 0.01. Also stop after a fixed number of steps, in case the slope never gets that small. Both checks go in the loop condition.

# Walk downhill on (x - 3) squared until the ground is nearly flat.
def slope(x):
    return 2 * (x - 3)

x = 0.0
rate = 0.1
steps = 0
while abs(slope(x)) > 0.01 and steps < 1000:
    x = x - rate * slope(x)
    steps = steps + 1
print(steps, round(x, 3))
# Change rate to 0.5, then to 0.01. How do the steps change?

A tiny rate crawls: many steps for a short walk. A good rate settles in a few dozen steps. A rate that is too big jumps over the bottom and lands higher up on the other side.

For x squared the slope is 2x. With a rate of 1.1 each step multiplies x by -1.2. The point flips sides and every jump is bigger than the last. This is divergence.

docker exec jupyter python3 -c "exec('x = 5.0\nfor n in range(1, 6):\n    x = x - 1.1 * 2 * x\n    print(n, round(x, 3))')"

In your own notebook, change the code block so the bowl is (x - 3) ** 2 + 10.

Predict first whether the steps and the final x change. The slope is the same, so only the height of the bowl moved.