Correlation is not causation

Correlation is a number from -1 to 1 that measures how closely two measurements rise and fall together. It is easy to find, and easy to misread.

When two things move together, it is tempting to say one causes the other. This lesson shows how to measure the link and why that is not enough.

On a scatter plot each day is one dot, with temperature across and hires up. If the dots climb from left to right, the link is positive. If they fall, it is negative.

The correlation coefficient, written r, puts that in one number. Its sign is the direction. Its size is the strength.

docker exec jupyter python3 -c "import pandas as pd; h = pd.read_csv('hires.csv'); print(round(h['temp_c'].corr(h['hires']), 2), round(h['rain_mm'].corr(h['hires']), 2))"

For each pair, find the temperature gap and the hires gap from their means. Multiply the two gaps. Add up the products. This is large and positive when high goes with high.

Then divide by the square root of the two sums of squared gaps. That scales the answer so it always lies between -1 and 1.

# Temperature in degrees and hires on five days.
temps = [2, 6, 10, 14, 18]
hires = [150, 190, 250, 260, 330]

mean_t = sum(temps) / len(temps)
mean_h = sum(hires) / len(hires)

together = 0
t_squares = 0
h_squares = 0
for n in range(len(temps)):
    t_gap = temps[n] - mean_t
    h_gap = hires[n] - mean_h
    together = together + t_gap * h_gap
    t_squares = t_squares + t_gap * t_gap
    h_squares = h_squares + h_gap * h_gap

r = together / (t_squares * h_squares) ** 0.5
print("correlation:", round(r, 3))
# Reverse the hires list: [330, 260, 250, 190, 150].
# What happens to the sign of r?

The coefficient measures how well a straight line fits. A strong curved pattern can give an r near zero.

Hires rise through spring and fall again after summer. A straight line through that hill is nearly flat, so the day of the year looks almost unrelated to hires. Always look at the scatter plot as well.

docker exec jupyter python3 -c "import pandas as pd; h = pd.read_csv('hires.csv'); print(round(h.index.to_series().corr(h['hires']), 2))"

Two things can move together for three reasons. One causes the other. Both are driven by a hidden third thing, a confounder. Or it is chance.

Bike hires and sunscreen sales both rise in a hot week. Sunscreen does not cause hires. The weather drives both. Only a controlled experiment, or strong knowledge of how things work, can show causation.

In your own notebook, draw the scatter plot of temp_c against hires, then find the correlation of temp_c with rain_mm.

It is close to zero. Say what that tells you about whether warm days are drier, and why a scatter plot still helps.