What if the server dies half way?
A shop takes your order: it reserves the stock, charges your card, emails a receipt and books a courier. What if the server dies just after the charge? Run the job again from the top and you pay twice. Durable execution is the way long jobs survive a crash: each finished step is written down, so a rerun carries on instead of starting again.
By the end of this lesson
Note
I can explain how a journal of finished steps lets a crashed workflow carry on without doing a step twice.
Key words
- Workflow: a job made of steps that must all happen, in order.
- Step: one piece of the job, such as charging a card.
- Journal: a table where each finished step and its result is written.
- Replay: reading a step's result from the journal instead of running it again.
- Idempotency key: a name for one request, so doing it twice has the effect of once.
- Saga: a workflow that undoes its finished steps when a later one fails.
Starter: what you already know
Note
Answer in your head. 1. What does a database table hold? 2. What happens to a program's variables when its process stops? 3. Why does a shop give each order a number? Answers: rows of data; they are lost; so it can find and talk about that one order.
Before you start
Note
Press Start the lab on the bar at the top of this lesson and wait until it says it is running. There are two containers. worker runs the Python and db is PostgreSQL. The terminal reaches them with docker exec, which runs one command inside a container. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.
A plain job starts from the top again
The first Run is the order job with no journal, told to crash just after charging the card. It prints two done lines and then the crash line. The second Run is the retry: it starts from the top and charges again. The third asks PostgreSQL for the payments of order-1, and there are 2 rows of 2500. The customer paid twice.
docker exec worker python3 naive.py order-1 --fresh --crash-after charge_cardShown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec worker python3 naive.py order-1Shown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec db psql -U durable -c "SELECT workflow, amount FROM payments WHERE workflow = 'order-1'"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
A journal remembers every finished step
order.py is the same order, written with the engine. After each step the engine writes the step and its result to the journal table. This Run crashes after 2 new steps. The journal then holds 2 rows: step 0 is reserve_stock and step 1 is charge_card, each with its result.
docker exec worker python3 order.py order-7 --fresh --crash-after 2Shown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec db psql -U durable -c "SELECT seq, step, result FROM journal WHERE workflow = 'order-7' ORDER BY seq"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Resuming replays the journal and carries on
Run the same workflow again, with no flags. It finds 2 journal rows, so it prints replay for steps 0 and 1 and takes their results from the rows. Only steps 2 and 3 really run. The card was charged once.
docker exec worker python3 order.py order-7Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Check
Tip
Why did the second run of order.py not charge the card again? Answer: step 1, charge_card, already had a row in the journal, so the engine replayed its result instead of running it.
An idempotency key makes a repeat harmless
There is one gap left: a crash after the charge but before its journal row. The first Run crashes exactly there. The rerun has no row for the charge, so it charges again. With --keyed the charge carries the key order-9:charge_card, and the card provider takes one payment per key: the second answer says "repeat": true and nothing more is taken.
docker exec worker python3 order.py order-9 --fresh --keyed --crash-inside charge_cardShown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec worker python3 order.py order-9 --keyedShown here, not run. Press Start lab to type it into your own lab’s terminal.
A saga undoes what was done
A trip needs a flight, a hotel and a car. This Run makes the car fail. The engine writes the failure down too, then runs the compensations newest first: cancel_hotel, then cancel_flight. The last line says nothing is left booked.
docker exec worker python3 saga.py trip-2 --fresh --fail-at book_carShown here, not run. Press Start lab to type it into your own lab’s terminal.
A common mistake
Warning
Many people think a journal alone means every effect happens exactly once. It does not: a crash between an effect and its journal row repeats that effect on the rerun. Exactly once needs both halves: the journal, so finished steps are not redone, and an idempotency key, so a repeated effect is ignored.
Exit check
Tip
Answer without looking back. 1. Where does the engine write a finished step? 2. What does a rerun do with a step that has a row? 3. Which crash can still repeat an effect? 4. What does a saga do when a step fails? Answers: in the journal table; replays its result; one after the effect and before its row; it undoes the finished steps, newest first.
For teachers
Tip
This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is two services, PostgreSQL and a Python worker holding an engine of about a hundred lines, and nothing here marks itself: the student hands in a workflow of their own, a crashed run, its resume and the journal rows. Key learning points: 1. A job with no journal repeats its steps after a crash. 2. A journal lets a rerun replay finished steps and carry on. 3. An idempotency key closes the gap between an effect and its row. Common mistake: thinking the journal alone gives exactly once. Equipment: the lab and its terminal. Tip: ask students to predict which lines will say replay before each rerun. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.