Why did the customer pay twice?
A long-running job is a program that does several steps in order, such as taking an order in a shop. When its process dies half way, a plain retry starts again from the top and repeats the steps already done.
That is how a customer is charged twice. This module builds the fix from first principles: durable execution, the idea behind workflow engines.
By the end of this lesson
Note
I can explain what goes wrong when a process dies in the middle of a job.
Key words
| Word | What it means |
|---|---|
| Process | a running program, with its own memory |
| Crash | a process stopping without warning |
| Side effect | a change outside the program, such as a payment |
| Retry | running a job again after it failed |
| Job | a piece of work a program does in several steps |
The cells students fill in are left blank in this preview.
A job is steps with effects outside
The lab's shop takes an order in four steps: reserve the stock, charge the card, email a receipt and book a courier. Most of them change something outside the program. Those changes are side effects.
A program's memory is lost when its process dies. A side effect is not. Once the card is charged, the money has left the customer's account, whatever the program forgets.
List the lab's files
docker exec worker lsShown here, not run. Press Start lab to type it into your own lab’s terminal.
Run the plain job and crash it
docker exec worker python3 naive.py order-1 --fresh --crash-after charge_cardShown here, not run. Press Start lab to type it into your own lab’s terminal.
A plain retry starts from the top
The first Run printed two done lines, reserve_stock and charge_card, then the crash line. The process stopped there, with exit code 3.
The next Run is the retry. The job has no record of what it did, so it starts from the top. It reserves, charges, emails and books. Then a query asks PostgreSQL how many payments order-1 has.
Run the plain job again
docker exec worker python3 naive.py order-1Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Count the payments for order-1
docker exec db psql -U durable -c "SELECT workflow, amount FROM payments WHERE workflow = 'order-1'"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: what each run did
| Run | Steps that finished | Payments so far |
|---|---|---|
| 1, crashed | reserve_stock, charge_card | 1 of 2500 |
| 2, the retry | all four steps | 2 of 2500 |
| What the customer wanted | all four steps, once | 1 of 2500 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a retry is always safe because the job failed. A retry is safe only when every step can run twice without harm. Charging a card cannot, so the retry charged the customer again.
Try it: move the crash
STEPS = ["reserve_stock", "charge_card", "send_receipt", "book_courier"]
EFFECTS = ["charge_card", "send_receipt"]
CRASH_AFTER = 2 # the first run dies after this many steps
done = []
# First run: it crashes part of the way through.
for name in STEPS[:CRASH_AFTER]:
done.append(name)
# The retry starts again from the top and finishes.
for name in STEPS:
done.append(name)
for name in EFFECTS:
print(name, "happened", done.count(name), "times")
# Change CRASH_AFTER to 3, then to 0, and run it again.
Where the crash lands decides the damage
A crash before any side effect costs nothing: the retry does the work once. A crash after the charge repeats the charge. A crash after the receipt repeats the charge and the email.
You cannot choose where a crash lands. Power fails, a server is replaced, a deploy restarts the process. So the job itself must remember what it has done. Lesson 2 gives it a memory that outlives the process.