How do you test a claim with data?
A hypothesis test asks whether a pattern in data could just be chance. It gives a number, the p-value, that says how surprising the pattern would be if nothing real were going on.
You need it because every data set shows some pattern, and many of those patterns are not real.
By the end of this lesson
Note
I can test a claim and avoid the common mistakes.
Key words
| Word | What it means | Example |
|---|---|---|
| Null hypothesis | the plain claim that nothing is going on | |
| Test statistic | one number that sums up the pattern you saw | |
| Permutation | a shuffle of which group each value belongs to | |
| p-value | the chance of a pattern this strong if only chance is at work | |
| Significant | a p-value below the line you chose beforehand | p below 0.05 |
The cells students fill in are left blank in this preview.
A test starts by assuming nothing happens
Say a manager claims that weekends are busier than weekdays. You do not try to prove it. You first assume the null hypothesis: weekends and weekdays are the same, and any gap in the data is luck.
Then you ask how likely a gap as big as yours would be under that assumption. You sum up the pattern in one number, the test statistic.
Worked example: the pieces of a test
| Piece | In words | For the bike claim |
|---|---|---|
| The claim | What you suspect | Weekends are busier |
| The null hypothesis | Nothing is going on | Weekend and weekday hires have the same mean |
| The test statistic | One number for the pattern | Weekend mean minus weekday mean |
| The p-value | How surprising the number is | Worked out next |
The cells students fill in are left blank in this preview.
Find the weekend gap in the hires
docker exec jupyter python3 -c "import pandas as pd; h = pd.read_csv('hires.csv'); w = h['weekday'].isin(['Sat', 'Sun']); print(round(h[w]['hires'].mean() - h[~w]['hires'].mean(), 1))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Find the gap between Mondays and Tuesdays
docker exec jupyter python3 -c "import pandas as pd; h = pd.read_csv('hires.csv'); m = h[h['weekday'] == 'Mon']['hires'].mean(); t = h[h['weekday'] == 'Tue']['hires'].mean(); print(round(m - t, 1))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Shuffle the labels to see what chance does
If weekend and weekday made no difference, the labels would not matter. You could swap them at random and the gap would be about as big.
So do exactly that. Shuffle which days count as weekend. Work out the gap. Repeat many times. Count how often a shuffle gives a gap as big as the real one. That share is the p-value.
Worked example: twelve days, all the splits
| Step | What happens | Result |
|---|---|---|
| Real data | 6 weekend days average 252.0, 6 weekdays average 181.2 | Gap 70.8 |
| Every possible split | Twelve days divided into two groups of six | 924 splits |
| Count the big gaps | Splits with a gap of 70.8 or more | 22 splits |
| The p-value | 22 divided by 924 | 0.024 |
The cells students fill in are left blank in this preview.
Shuffle the twelve days in Python
import random
# Hires on the first six weekend days and the first six weekdays.
weekend = [327, 204, 244, 281, 185, 271]
weekday = [137, 212, 173, 101, 243, 221]
def mean(values):
return sum(values) / len(values)
seen = mean(weekend) - mean(weekday)
print("the real gap:", round(seen, 1))
both = weekend + weekday
random.seed(1)
as_big = 0
for go in range(1000):
random.shuffle(both)
gap = mean(both[:6]) - mean(both[6:])
if gap >= seen:
as_big = as_big + 1
print("shuffles with a gap as big, out of 1000:", as_big)
# Change 327 to 127 and run again. What happens to the count?
Shuffle the whole year 2000 times
docker exec jupyter python3 -c "import numpy as np, pandas as pd; h = pd.read_csv('hires.csv'); v = h['hires'].to_numpy(); w = h['weekday'].isin(['Sat', 'Sun']).to_numpy(); gap = lambda x: x[w].mean() - x[~w].mean(); g = np.random.default_rng(1); print(sum(gap(g.permutation(v)) >= gap(v) for _ in range(2000)))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What a p-value does and does not say
A p-value says how surprising your data is if the null hypothesis is true. A small one, often below 0.05, means chance alone rarely gives a gap like yours. People call that significant.
It does not say how big or important the gap is. It does not say the claim is true. A p-value above the line does not show there is no effect, only that this data cannot tell.
Worked example: many tests find false alarms
| Tests run on data with no real pattern | Chance of at least one p below 0.05 |
|---|---|
| 1 | 0.05 |
| 5 | 0.23 |
| 10 | 0.40 |
| 20 | 0.64 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think p = 0.03 means a 3% chance the claim is wrong. It does not. It means that if nothing were going on, 3 in 100 shuffles would give a gap this big. Run twenty tests and one will usually look significant by luck, so say how many you ran.
Going further
In your own notebook, test whether sunny days hire more than cloudy days. Write down the null hypothesis, the test statistic and what you expect before you shuffle.
Then change the number of shuffles from 2000 to 200. Does the answer change? Explain why a larger number of shuffles gives a steadier p-value.