Daily visits to the school library
A librarian counted visits each day for seven days. The list has an empty entry, a word instead of a number and one day with 300, when a whole year group came in.
Is footfall growing? You can only say after you have cleaned, looked for outliers and fitted a line.
Key words
| Word | What it means |
|---|---|
| Interrogate | ask the data questions by sorting, filtering and counting |
| Clean | remove or fix values that cannot be used |
| Outlier | a value far away from the rest of the data |
| Skewed data | data that bunches at one end with a long tail at the other |
| Regression | finding the line that best describes how one value changes with another |
| Median | the middle value when the data is in order |
| Dataset | a collection of records held together for analysis |
The cells students fill in are left blank in this preview.
By the end of this lesson
Note
I can take a messy dataset through interrogating, cleaning, handling outliers and skew, and regression, and say what it shows.
Starter
Have a go
Starter
3 questionsYour answers are marked here and are not kept. A college account keeps every student’s marks.
The raw readings
| Day | Raw entry | Problem |
|---|---|---|
| 1 | 10 | None |
| 2 | 12 | None |
| 3 | (empty) | Incomplete value |
| 4 | x | Not a number |
| 5 | 14 | None |
| 6 | 300 | Outlier: a whole year group visited |
| 7 | 16 | None |
The cells students fill in are left blank in this preview.
Worked example: what each step leaves
| Step | What happens | Values left |
|---|---|---|
| Raw | Seven entries | 10, 12, empty, x, 14, 300, 16 |
| Clean | Drop the empty one and x | 10, 12, 14, 300, 16 |
| Outliers | The median is 14, so 300 is far away. Drop it and note why. | 10, 12, 14, 16 |
| Skew | Report the median of what is left, 13 | 10, 12, 14, 16 |
| Regression | Slope against day 1, 2, 3, 4 is 2 | Visits rise by about 2 a day |
The cells students fill in are left blank in this preview.
Try it: the first three steps
readings = ["10", "12", "", "x", "14", "300", "16"]
numbers = []
for text in readings:
if text.isdigit():
numbers.append(int(text))
print("cleaned", numbers)
ordered = sorted(numbers)
middle = ordered[len(ordered) // 2]
print("median", middle)
kept = []
for n in numbers:
if abs(n - middle) <= 100:
kept.append(n)
print("kept", kept)
Say what you did
The finished story has numbers and reasons: "Out of 7 entries, 4 were usable. One was dropped as an empty value, one as a word, and one as an outlier, because a year group visited. The median is 13. Visits rise by about 2 a day."
Say what you removed and why, so someone else can check.