Mean, variance and standard deviation

A standard deviation is a number that says how far values usually sit from their mean. You need it because two data sets can share a mean and still behave very differently.

Two bike stations can both hire 240 bikes a day on average. One may hire close to 240 every day. The other may swing between 100 and 380.

The mean tells you where the middle of the data is. It says nothing about how far the values stray from it. That second fact is the spread.

The simplest measure of spread is the range: the highest value minus the lowest.

The range uses only two days, the highest and the lowest. One odd day can change it a lot.

A quartile cuts ordered data into quarters. The first quartile has a quarter of the values below it. The third has three quarters below it. The gap between them is the interquartile range. It is the spread of the middle half of the days.

docker exec jupyter python3 -c "import pandas as pd; h = pd.read_csv('hires.csv'); print(h['hires'].min(), h['hires'].max(), h['hires'].quantile(0.25), h['hires'].quantile(0.75))"

A better spread uses every value. For each value, find its gap from the mean. Square each gap, so that a gap below the mean does not cancel a gap above it. Then take the mean of the squares. That is the variance.

The variance is in squared units, such as bikes squared. Its square root, the standard deviation, is back in bikes.

# Five days of hires at one station.
days = [180, 210, 240, 260, 310]

total = 0
for one in days:
    total = total + one
centre = total / len(days)

squares = 0
for one in days:
    gap = one - centre
    squares = squares + gap * gap
print("mean:", centre)
print("variance:", squares / len(days))
print("standard deviation:", round((squares / len(days)) ** 0.5, 1))
# Change 310 to 500 and run again.
# Which of the three numbers grows the most?

When your data is every value there is, divide the squared gaps by how many values you have, n. That gives the population variance.

When your data is only a sample of a bigger group, divide by n - 1 instead. A sample sits closer to its own mean than the whole group sits to the true mean, and the smaller divisor makes up for it.

docker exec jupyter python3 -c "import numpy as np, pandas as pd; v = pd.read_csv('hires.csv')['hires'].head(5); print(round(v.std(), 1), round(np.std(v.to_numpy()), 1))"

In your own notebook, find the standard deviation of temp_c, then of rain_mm. Which one varies more in its own unit?

Then find a pair of columns where the standard deviations are close but the units differ. Say why comparing them directly would mislead a reader.