Kinds of day nobody named
The shop wants three staff rotas: one for slow days, one for ordinary days and one for rush days. Nobody has ever marked a day as slow or rush.
There is no answer column to learn from. The days have to sort themselves into groups, and you have to judge whether the groups make sense.
By the end of this lesson
Note
I can group unlabelled data with k-means.
Key words
| Word | What it means |
|---|---|
| Unlabelled data | rows with no answer column |
| Cluster | a group of rows close to each other |
| Centroid | the mean point of a cluster |
| k-means | a method that moves k centroids until the groups stop changing |
| Scaling | giving every feature the same spread before measuring distance |
| Inertia | the total squared distance from each point to its centroid |
The cells students fill in are left blank in this preview.
k-means repeats two steps
Choose k, the number of groups. Place k centres somewhere among the points.
Then repeat two steps. Give every point to its nearest centre. Move every centre to the mean of its points, which makes it a centroid. Stop when no point changes group. Different starting places can end in different groups, so scikit-learn tries several and keeps the best.
One step by hand
# One step of k-means on six days, by hires alone.
days = [120, 150, 180, 300, 330, 420]
centres = [100, 400]
groups = [[], []]
for one in days:
if abs(one - centres[0]) <= abs(one - centres[1]):
groups[0].append(one)
else:
groups[1].append(one)
print("groups:", groups)
for n in range(2):
centres[n] = sum(groups[n]) / len(groups[n])
print("new centres:", centres)
# Run the step again from the new centres.
# Does any day change group?
Scale before you measure distance
Hires run from 41 to 487 a day. Temperatures run from about -1 to 25 degrees. A gap of 10 hires is tiny; a gap of 10 degrees is huge.
But a distance treats both 10s the same. So without scaling, hires decide everything. StandardScaler gives each feature a mean of 0 and the same spread, so each counts fairly.
Type this in your notebook: no scaling
import pandas as pd
from sklearn.cluster import KMeans
hires = pd.read_csv("hires.csv")
# No scaling: hires spread over hundreds, temperatures over about 26.
hires["raw"] = KMeans(n_clusters=3, n_init=10, random_state=0).fit_predict(hires[["temp_c", "hires"]])
hires.groupby("raw")[["temp_c", "hires"]].agg(["min", "max"])
Group the days, scaled
docker exec jupyter python3 steps/kmeans.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think k-means discovers how many groups the data really has. It always returns exactly the k groups you asked for, even from data with no groups at all. Choosing k, and naming each group, is your judgement.
The elbow helps you choose k
The inertia is the total squared distance from each point to its centroid. More groups always lower it, and with one group a point each it reaches 0.
So look for the elbow: the k after which another group buys little. Then ask whether the groups are useful to the shop.
Spread for k from 1 to 6
docker exec jupyter python3 steps/elbow.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Type this in your notebook: see the groups
from sklearn.preprocessing import StandardScaler
scaled = StandardScaler().fit_transform(hires[["temp_c", "hires"]])
hires["group"] = KMeans(n_clusters=3, n_init=10, random_state=0).fit_predict(scaled)
# One colour for each group.
hires.plot.scatter(x="temp_c", y="hires", c="group", colormap="viridis")
Going further
In your notebook, cluster on three features: temperature, rain and hires, all scaled. Try k = 3 and k = 4.
For each group, print its size and its mean of each feature. Give each group a name a rota planner would understand, and say which k you would choose and why.