The messy sign-up sheet
Your class signs up for a trip in the group chat. Three people type their details. Look:
" sam patel ,16, ls1 1ab , £4.50 " "AISHA KHAN,160,m60 1nw,£3.00" "leo brown,14,ls2 9xy,£2"
The capital letters are all over the place. There are extra spaces. One age is 160. You cannot send this to the bus company. First you must tidy it.
Key words
| Word | What it means | Example |
|---|---|---|
| Data wrangling | Turning messy raw data into tidy data that is ready to use. | Fixing the sign-up sheet |
| Structure | Put data into rows and fields so a computer can read it. | Split a line at each comma. |
| Clean | Fix or remove mistakes, extra spaces and odd capital letters. | "sam patel" becomes "SAM PATEL" |
| Validate | Check that each value is sensible. Remove or flag the rest. | Age 160 is refused. |
| Enrich | Add useful new facts to the data. | Add a ticket type. |
| Output | Save or send the tidy data in the form the next step needs. | A CSV file |
| Field | One piece of information in a record, such as the age. | age |
| CSV | A text file where each line is a row and commas split the fields. | Sam,16,LS1 1AB |
The cells students fill in are left blank in this preview.
What is data wrangling, and why?
Data wrangling turns messy raw data into tidy data. Raw data from people, forms, machines and other systems is rarely ready to use.
Wrangle data before you analyse it or put it in a store. Bad data in gives bad answers out. Use it when the data comes from many places or many people.
Quick check: what is wrong?
Have a go
Quick check: what is wrong?
3 questionsYour answers are marked here and are not kept. A college account keeps every student’s marks.
Before and after
| Raw data | Problem | Tidy data |
|---|---|---|
| sam patel ,16, ls1 1ab | Spaces and small letters | SAM PATEL,16,LS1 1AB |
| AISHA KHAN,160,m60 1nw | Age is not sensible | Not kept |
| leo brown,14,ls2 9xy | Small letters | LEO BROWN,14,LS2 9XY |
The cells students fill in are left blank in this preview.
Steps 1 and 2: structure and clean
Structure: give the data a shape. Split each line at the commas, so every value has its own place.
Clean: fix small problems. Take off extra spaces. Write names in capital letters. Turn "£4.50" into the number 4.5.
Try it: structure and clean
line = " sam patel ,16, ls1 1ab , £4.50 "
fields = []
for part in line.split(","):
fields.append(part.strip())
print("Structured:", fields)
name = fields[0].upper()
postcode = fields[2].upper()
price = float(fields[3].replace("£", ""))
print("Cleaned:", name, postcode, price)
# Change the line. Add a new person and press Run again.
Steps 3, 4 and 5
Validate: check each value is sensible. Age 160 goes.
Enrich: add something useful. A ticket type, a town from the postcode, or the day of the week from a date.
Output: save the tidy data in the shape the next step needs, such as a CSV file.