Two questions about a year of hires
A bike hire shop keeps one row for every day of the year. The owner asks two questions. Are weekends really busier? And how many days were very busy?
Scrolling through 365 rows will not answer either of them. A few lines of pandas will, in seconds.
By the end of this lesson
Note
I can load a table of data and ask questions of it.
Key words
| Word | What it means | Example |
|---|---|---|
| DataFrame | a table in pandas, with named columns | |
| Column | one named field, the same for every row | hires["temp_c"] |
| Filter | keep only the rows that pass a test | hires["hires"] > 400 |
| Group | split the rows by the value in one column | groupby("weather") |
| Summarise | turn many values into one, such as a mean | |
| Median | the middle value once the values are in order | |
| Mode | the value that appears most often |
The cells students fill in are left blank in this preview.
Two ways into the lab
Press Start on the lesson bar and wait until the lab is running. The terminal beside it has no Python of its own. Each Run button types docker exec jupyter, which runs one command inside the lab's container.
Open JupyterLab from the bar for the notebook. A block titled Type this in your notebook goes in a new notebook cell. It will not run on this page, because the page's Python has no pandas.
A DataFrame is a table with named columns
pd.read_csv reads a CSV file into a DataFrame. Each column has a name from the first line of the file.
Square brackets with a name pick one column: hires["temp_c"] is every day's temperature. A column has its own methods, such as max, mean and value_counts.
Look at the raw file
docker exec jupyter head -n 4 hires.csvShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Type this in your notebook: one column
import pandas as pd
hires = pd.read_csv("hires.csv")
# One column: a temperature for each day.
hires["temp_c"].max()
A filter keeps rows that pass a test
hires["hires"] > 400 asks a question of every row. The answer is a column of True and False.
Put that answer inside hires[...] and pandas keeps only the True rows. To join two tests, write & for "and" and | for "or". Put each test in its own brackets.
Count the very busy days
docker exec jupyter python3 steps/filter.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Type this in your notebook: two tests
# Busy days that were also sunny.
busy_sunny = hires[(hires["hires"] > 400) & (hires["weather"] == "sunny")]
len(busy_sunny)
A common mistake
Warning
Many people think Python's and joins two tests in pandas. It does not: and wants one True or False, and a test on a column gives 365 of them. pandas stops with "The truth value of a Series is ambiguous". Use & and brackets round each test.
groupby splits the rows, then summarises
hires.groupby("weekday")["hires"].mean() does three things. It splits the rows by weekday. It picks the hires column in each group. It takes the mean of each.
The mean is pulled by a few extreme days. The median is not, so look at both. The mode suits a column of words, such as the weather.
Worked example: one big day
| Days | Mean | Median | Mode |
|---|---|---|---|
| 204, 173, 327, 173, 240, 198, 487 | 1802 / 7 = 257.4 | 204, the 4th of 7 in order | 173, seen twice |
| The same without 487 | 1315 / 6 = 219.2 | (198 + 204) / 2 = 201 | 173, seen twice |
The cells students fill in are left blank in this preview.
Mean and median by hand
# Seven days of hires at one station.
days = [204, 173, 327, 173, 240, 198, 487]
total = 0
for one in days:
total = total + one
print("mean:", round(total / len(days), 1))
in_order = sorted(days)
middle = len(in_order) // 2
print("median:", in_order[middle])
# Take 487 out of the list and run again.
# Which of the two numbers moves more?
Compare weekends with weekdays
docker exec jupyter python3 steps/weekend.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Type this in your notebook: by weekday
# The middle day for each weekday, Monday first.
order = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
hires.groupby("weekday")["hires"].median().reindex(order)
Going further
In your notebook, find the quietest month. Group by month, take the mean of hires, and use sort_values to put it in order.
Then ask a question of your own with a filter and a group together. For example: on rainy days only, are weekends still busier? Say what you expect before you run it.