Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

What goes wrong when a job crashes

After this lesson: I can explain what goes wrong when a process dies in the middle of a job.

60 min lesson4 commandsDurable execution lab: crash a workflow on purpose and resume it from its journal

Why did the customer pay twice?

A long-running job is a program that does several steps in order, such as taking an order in a shop. When its process dies half way, a plain retry starts again from the top and repeats the steps already done.

That is how a customer is charged twice. This module builds the fix from first principles: durable execution, the idea behind workflow engines.

By the end of this lesson

Note

I can explain what goes wrong when a process dies in the middle of a job.

Key words

WordWhat it means
Processa running program, with its own memory
Crasha process stopping without warning
Side effecta change outside the program, such as a payment
Retryrunning a job again after it failed
Joba piece of work a program does in several steps

The cells students fill in are left blank in this preview.

A job is steps with effects outside

The lab's shop takes an order in four steps: reserve the stock, charge the card, email a receipt and book a courier. Most of them change something outside the program. Those changes are side effects.

A program's memory is lost when its process dies. A side effect is not. Once the card is charged, the money has left the customer's account, whatever the program forgets.

List the lab's files

Command
docker exec worker ls

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Run the plain job and crash it

Command
docker exec worker python3 naive.py order-1 --fresh --crash-after charge_card

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A plain retry starts from the top

The first Run printed two done lines, reserve_stock and charge_card, then the crash line. The process stopped there, with exit code 3.

The next Run is the retry. The job has no record of what it did, so it starts from the top. It reserves, charges, emails and books. Then a query asks PostgreSQL how many payments order-1 has.

Run the plain job again

Command
docker exec worker python3 naive.py order-1

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Count the payments for order-1

Command
docker exec db psql -U durable -c "SELECT workflow, amount FROM payments WHERE workflow = 'order-1'"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: what each run did

RunSteps that finishedPayments so far
1, crashedreserve_stock, charge_card1 of 2500
2, the retryall four steps2 of 2500
What the customer wantedall four steps, once1 of 2500

The cells students fill in are left blank in this preview.

A common mistake

Warning

Many people think a retry is always safe because the job failed. A retry is safe only when every step can run twice without harm. Charging a card cannot, so the retry charged the customer again.

Try it: move the crash

Code: python
STEPS = ["reserve_stock", "charge_card", "send_receipt", "book_courier"]
EFFECTS = ["charge_card", "send_receipt"]
CRASH_AFTER = 2   # the first run dies after this many steps

done = []
# First run: it crashes part of the way through.
for name in STEPS[:CRASH_AFTER]:
    done.append(name)
# The retry starts again from the top and finishes.
for name in STEPS:
    done.append(name)

for name in EFFECTS:
    print(name, "happened", done.count(name), "times")
# Change CRASH_AFTER to 3, then to 0, and run it again.

Where the crash lands decides the damage

A crash before any side effect costs nothing: the retry does the work once. A crash after the charge repeats the charge. A crash after the receipt repeats the charge and the email.

You cannot choose where a crash lands. Power fails, a server is replaced, a deploy restarts the process. So the job itself must remember what it has done. Lesson 2 gives it a memory that outlives the process.

About this lab and its source: Durable execution lab: crash a workflow on purpose and resume it from its journal

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.