A decision built on sand
A club counts its members from a table. The table counts Sam twice and ignores everyone with a blank age. The club thinks it has 40 members. It has 31.
It orders too many snacks and charges the wrong fee. Poor quality data leads to poor decisions.
Key words
| Word | What it means | Example |
|---|---|---|
| Cleansing | Fixing or removing poor data so it can be trusted. | Deleting a repeated row |
| Bias | A leaning that makes results unfair or untrue for some groups. | Only asking people who are in a good mood |
| Remove | To take out rows or values that should not be there. | Duplicates, old data, outliers |
| Edit | To correct a value. | Fix the typo Brwn to Brown |
| Parse | To read text and split it into a form a program can use. | "Sam:15" into a name and an age |
| Confidence | How far you can trust the data. | High after checking |
| Typo | A small typing mistake. | Brwn |
The cells students fill in are left blank in this preview.
By the end of this lesson
Note
I can say why fixing quality matters, and use remove, edit and parse to build confidence in data.
Starter: what you already know
Have a go
Starter: what you already know
3 questionsYour answers are marked here and are not kept. A college account keeps every student’s marks.
Two reasons quality matters
First, clean data gives accurate and meaningful results. A wrong count gives a wrong decision.
Second, it helps you avoid bias in how data is managed. If you keep only the easy rows, the data speaks for some people and not for others. Fair results need fair data.
Three ways to cleanse
| Action | What it does | Example |
|---|---|---|
| Remove | Takes out what should not be there | Duplicates, out-of-date rows, outliers |
| Edit | Corrects errors | Fix a typo |
| Parse | Reads text into fields | Split "Sam:15" into name and age |
The cells students fill in are left blank in this preview.
Try it: remove, edit, parse
names = ["Sam", "Jo", "Sam"]
unique = []
for name in names:
if name not in unique:
unique.append(name)
print("Removed repeats:", unique)
pair = "Sam:15".split(":")
print("Parsed:", pair[0], int(pair[1]) + 1)
# Try it: add "Jo" to the end of names and press Run.
A common mistake
Warning
Many people delete an outlier just because it is large. A big value may be real. One very big order could be a real customer. Check it first. Remove it only if it is an error.