What is a probability distribution?
A probability distribution shows how likely each possible value of a measurement is. Two shapes turn up again and again: the binomial, for counting successes, and the normal, the bell curve.
Once you know which shape your data has, you can say how surprising a value is without checking every row.
By the end of this lesson
Note
I can read a normal and a binomial distribution.
Key words
| Word | What it means | Example |
|---|---|---|
| Distribution | how often each value happens | |
| Trial | one go at something that can succeed or fail | one day, rainy or not |
| Binomial | the count of successes in a fixed number of trials | |
| Normal | a bell-shaped distribution set by its mean and spread | |
| Z-score | how many standard deviations a value is from the mean |
The cells students fill in are left blank in this preview.
A binomial counts trials that succeed
Use the binomial when four things hold. The number of trials is fixed. Each trial has two results. The chance of success is the same each time. The trials do not affect each other.
To find the chance of exactly k successes, count the ways to choose which trials succeed. Multiply by the chance of each success and each failure.
Worked example: rain on four days
| Rainy days | Ways to choose them | Working | Chance |
|---|---|---|---|
| 0 | 1 | 0.7 x 0.7 x 0.7 x 0.7 | 0.2401 |
| 1 | 4 | 4 x 0.3 x 0.7 x 0.7 x 0.7 | 0.4116 |
| 2 | 6 | 6 x 0.3 x 0.3 x 0.7 x 0.7 | 0.2646 |
| 3 | 4 | 4 x 0.3 x 0.3 x 0.3 x 0.7 | 0.0756 |
| 4 | 1 | 0.3 x 0.3 x 0.3 x 0.3 | 0.0081 |
| All | 1.0000 |
The cells students fill in are left blank in this preview.
Count rainy days in blocks of five
docker exec jupyter python3 -c "import numpy as np, pandas as pd; h = pd.read_csv('hires.csv'); r = h['weather'].eq('rainy').astype(int).groupby(h.index // 5).sum(); print(np.bincount(r, minlength=6).tolist())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: real blocks against the binomial
| Rainy days in a block of five | Blocks seen | Blocks a binomial expects |
|---|---|---|
| 0 | 11 | 9.9 |
| 1 | 21 | 24.4 |
| 2 | 29 | 23.9 |
| 3 | 8 | 11.7 |
| 4 | 3 | 2.9 |
| 5 | 1 | 0.3 |
The cells students fill in are left blank in this preview.
Real counts follow the shape loosely
The command splits the year into 73 blocks of five days and counts the rainy days in each. It printed [11, 21, 29, 8, 3, 1].
The right-hand column of the table is the binomial with the year's share of rainy days, 120 of 365. The real counts follow its shape. Chance makes them bumpy, so 29 blocks with two rainy days is a little above the 23.9 expected.
The normal distribution is a bell
The normal distribution is a symmetric bell. The mean says where its middle is. The standard deviation says how wide it is.
A rule of thumb saves a lot of work. About 68 in 100 values lie within one standard deviation of the mean. About 95 lie within two. Almost all lie within three.
Worked example: the 68 and 95 rule
| Distance from the mean | Rule says about | Days in the lab data |
|---|---|---|
| Within 1 standard deviation | 68 of 100 | 241 of 365 (66 of 100) |
| Within 2 standard deviations | 95 of 100 | 355 of 365 (97 of 100) |
| Within 3 standard deviations | Almost all | 365 of 365 |
The cells students fill in are left blank in this preview.
Count days within one and two deviations
docker exec jupyter python3 -c "import pandas as pd; v = pd.read_csv('hires.csv')['hires']; m = v.mean(); s = v.std(); print(((v - m).abs() <= s).sum(), ((v - m).abs() <= 2 * s).sum())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A z-score says how unusual a value is
A z-score is a value minus the mean, divided by the standard deviation. It counts how many standard deviations the value is from the middle.
A z-score near 0 is a typical day. A z-score beyond 2 or below -2 is unusual: about 1 value in 20.
Work out z-scores in Python
# How far is a day from the typical day?
mean = 270.3
spread = 79.7
for hires in [150, 300, 400, 487]:
z = (hires - mean) / spread
print(hires, "hires is", round(z, 2), "standard deviations from the mean")
# Add 41, the quietest day of the year, to the list.
# Is it further from the mean than 487?
A common mistake
Warning
Many people think all data forms a bell curve. It does not. Waiting times and incomes pile up on one side. Draw a histogram before you use the 68, 95 rule, and only use it when the shape is roughly a symmetric bell.
Going further
In your own notebook, work out the share of days within one standard deviation of the mean for temp_c. Compare it with 68 in 100.
Then draw a histogram of rain_mm. Say why the 68, 95 rule would be a poor fit for it.