From source to report: pipelines

A shop chain's managers open a sales report at 7 every morning. Nobody built it by hand overnight.

The tills recorded each sale. A program copied the sales, tidied them and stored them. The report read the stored rows.

That chain of steps is a pipeline. Building and fixing pipelines is the daily work of a data engineer.

Extract copies the data out of the source. The till system stays as it was.

Transform cleans and reshapes the copy. It tidies names, turns text into numbers and leaves out bad rows.

Load writes the clean rows into a store built for reports. The report reads only that store.

Real pipelines run in tools built for the job. Underneath, each step is still a small program.

The block below is a whole pipeline in a few lines. Find the three steps by their comments. Then run it and count the rows that reach the store.

# EXTRACT: the till system exports text, one sale a line.
export = """2026-10-01,Leeds, 12.50
2026-10-01,york,8.00
2026-10-01,Leeds,
2026-10-01,York ,4.50"""

# TRANSFORM: split each line, tidy the city, skip a sale with no amount.
clean = []
for line in export.split("\n"):
    parts = line.split(",")
    if parts[2].strip() == "":
        continue
    clean.append({"day": parts[0], "city": parts[1].strip().title(),
                  "amount": float(parts[2])})

# LOAD: add the clean rows to the store the report reads.
store = []
for row in clean:
    store.append(row)

print("Rows loaded:", len(store))
for row in store:
    print(row)

# Add a line "2026-10-01,hull,3.20" to the export, then press Run.