Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Durable Execution Basics

The whole lesson is free and interactive. Read it here, or press Start lab to run it for real in a live lab, with no account needed.

Container lab11 min read9 commands

What if the server dies half way?

A shop takes your order: it reserves the stock, charges your card, emails a receipt and books a courier. What if the server dies just after the charge? Run the job again from the top and you pay twice. Durable execution is the way long jobs survive a crash: each finished step is written down, so a rerun carries on instead of starting again.

By the end of this lesson

Note

I can explain how a journal of finished steps lets a crashed workflow carry on without doing a step twice.

Key words

  • Workflow: a job made of steps that must all happen, in order.
  • Step: one piece of the job, such as charging a card.
  • Journal: a table where each finished step and its result is written.
  • Replay: reading a step's result from the journal instead of running it again.
  • Idempotency key: a name for one request, so doing it twice has the effect of once.
  • Saga: a workflow that undoes its finished steps when a later one fails.

Starter: what you already know

Note

Answer in your head. 1. What does a database table hold? 2. What happens to a program's variables when its process stops? 3. Why does a shop give each order a number? Answers: rows of data; they are lost; so it can find and talk about that one order.

Before you start

Note

Press Start the lab on the bar at the top of this lesson and wait until it says it is running. There are two containers. worker runs the Python and db is PostgreSQL. The terminal reaches them with docker exec, which runs one command inside a container. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.

A plain job starts from the top again

The first Run is the order job with no journal, told to crash just after charging the card. It prints two done lines and then the crash line. The second Run is the retry: it starts from the top and charges again. The third asks PostgreSQL for the payments of order-1, and there are 2 rows of 2500. The customer paid twice.

Command
docker exec worker python3 naive.py order-1 --fresh --crash-after charge_card

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
docker exec worker python3 naive.py order-1

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
docker exec db psql -U durable -c "SELECT workflow, amount FROM payments WHERE workflow = 'order-1'"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A journal remembers every finished step

order.py is the same order, written with the engine. After each step the engine writes the step and its result to the journal table. This Run crashes after 2 new steps. The journal then holds 2 rows: step 0 is reserve_stock and step 1 is charge_card, each with its result.

Command
docker exec worker python3 order.py order-7 --fresh --crash-after 2

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
docker exec db psql -U durable -c "SELECT seq, step, result FROM journal WHERE workflow = 'order-7' ORDER BY seq"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Resuming replays the journal and carries on

Run the same workflow again, with no flags. It finds 2 journal rows, so it prints replay for steps 0 and 1 and takes their results from the rows. Only steps 2 and 3 really run. The card was charged once.

Command
docker exec worker python3 order.py order-7

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Check

Tip

Why did the second run of order.py not charge the card again? Answer: step 1, charge_card, already had a row in the journal, so the engine replayed its result instead of running it.

An idempotency key makes a repeat harmless

There is one gap left: a crash after the charge but before its journal row. The first Run crashes exactly there. The rerun has no row for the charge, so it charges again. With --keyed the charge carries the key order-9:charge_card, and the card provider takes one payment per key: the second answer says "repeat": true and nothing more is taken.

Command
docker exec worker python3 order.py order-9 --fresh --keyed --crash-inside charge_card

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
docker exec worker python3 order.py order-9 --keyed

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A saga undoes what was done

A trip needs a flight, a hotel and a car. This Run makes the car fail. The engine writes the failure down too, then runs the compensations newest first: cancel_hotel, then cancel_flight. The last line says nothing is left booked.

Command
docker exec worker python3 saga.py trip-2 --fresh --fail-at book_car

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A common mistake

Warning

Many people think a journal alone means every effect happens exactly once. It does not: a crash between an effect and its journal row repeats that effect on the rerun. Exactly once needs both halves: the journal, so finished steps are not redone, and an idempotency key, so a repeated effect is ignored.

Exit check

Tip

Answer without looking back. 1. Where does the engine write a finished step? 2. What does a rerun do with a step that has a row? 3. Which crash can still repeat an effect? 4. What does a saga do when a step fails? Answers: in the journal table; replays its result; one after the effect and before its row; it undoes the finished steps, newest first.

For teachers

Tip

This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is two services, PostgreSQL and a Python worker holding an engine of about a hundred lines, and nothing here marks itself: the student hands in a workflow of their own, a crashed run, its resume and the journal rows. Key learning points: 1. A job with no journal repeats its steps after a crash. 2. A journal lets a rerun replay finished steps and carry on. 3. An idempotency key closes the gap between an effect and its row. Common mistake: thinking the journal alone gives exactly once. Equipment: the lab and its terminal. Tip: ask students to predict which lines will say replay before each rerun. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.

About this lab and its source: Durable execution lab: crash a workflow on purpose and resume it from its journal

Teaching a class? Set these lessons for your students with marking and a lab for every student.