Least squares line fitting

Least squares fitting finds the straight line that comes closest to a set of points. Every straight-line model in statistics and machine learning is fitted this way.

In this lesson you will measure how far a line is from the data, then use gradient descent to find the best line. You will check it against the exact answer.

A line hires = m * temp + c predicts hires from temperature. For each day, the residual is the prediction minus the real hires.

Positive and negative residuals would cancel, so square each one. The average of the squares is the mean squared error. A smaller number means a better line.

The mean squared error is a function of two inputs, m and c. Its gradient says how the error changes when each one moves.

For each day, the slope for m adds 2 times the residual times x. The slope for c adds 2 times the residual. Divide each total by the number of days, then step both numbers downhill.

# Temperature in tens of degrees, hires in hundreds.
xs = [0.2, 0.6, 1.0, 1.4, 1.8]
ys = [1.5, 1.9, 2.5, 2.6, 3.3]

m = 0.0
c = 0.0
rate = 0.1
for step in range(2000):
    slope_m = 0.0
    slope_c = 0.0
    for n in range(len(xs)):
        error = m * xs[n] + c - ys[n]
        slope_m = slope_m + 2 * error * xs[n] / len(xs)
        slope_c = slope_c + 2 * error / len(xs)
    m = m - rate * slope_m
    c = c - rate * slope_c
print(round(m, 3), round(c, 3))
# Change the 2000 to 20 and run it again.
# Is the line already close, or does it need more steps?

For a straight line you do not need descent. Find the mean of x and the mean of y. Multiply each pair of gaps and add them up. Add up the squared x gaps too.

The slope is the first total divided by the second. The intercept is the mean of y minus the slope times the mean of x. Descent matters because most models have no such formula.

docker exec jupyter python3 -c "import numpy as np; print(*np.polyfit([2, 6, 10, 14, 18], [150, 190, 250, 260, 330], 1).round(2))"

docker exec jupyter python3 -c "import numpy as np, pandas as pd; h = pd.read_csv('hires.csv'); print(*np.polyfit(h['temp_c'], h['hires'], 1).round(2))"

In your own notebook, fit a line of hires against rain_mm for the whole year with np.polyfit.

Say what the slope tells you about one more millimetre of rain. Then compare its sign with the temperature slope of 11.12.