How does a workflow survive a crash?
Durable execution writes each finished step of a workflow in a journal, so that a rerun can carry on after a crash. The journal is a table in a database, which outlives the process.
On a rerun, a step that has a row is replayed from it, not run again. Work starts at the first step with no row. The lab's engine does this in about a hundred lines.
By the end of this lesson
Note
I can use a step journal to find where a crashed workflow should carry on.
Key words
| Word | What it means |
|---|---|
| Workflow | a job whose steps must all happen, in order |
| Journal | a table holding each finished step and its result |
| Sequence number | a step's place in the workflow, counted from 0 |
| Replay | taking a step's result from the journal instead of running it |
| Resume | running a crashed workflow again so it carries on |
The cells students fill in are left blank in this preview.
The engine writes each result before going on
In the lab, a workflow is ordinary Python. Each step is a call to wf.step with a name and a function to run. The engine gives every step a sequence number: 0 for the first, 1 for the next.
When a step finishes, the engine inserts a row into the journal table: the workflow, the number, the name and the result. Only then does the workflow go on.
Crash the order workflow after two steps
docker exec worker python3 order.py order-7 --fresh --crash-after 2Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Read the journal rows for order-7
docker exec db psql -U durable -c "SELECT seq, step, result FROM journal WHERE workflow = 'order-7' ORDER BY seq"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
A resume replays the rows it finds
Now run the same workflow with no flags. The engine reads the journal first and finds two rows.
Steps 0 and 1 print replay: their results come from the rows, so nothing is reserved or charged again. Steps 2 and 3 have no rows, so they print run. Running it once more replays all four steps and does nothing new.
Resume order-7
docker exec worker python3 order.py order-7Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Resume order-7 once more
docker exec worker python3 order.py order-7Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: three runs of order-7
| Run | Rows found | Replayed | Ran |
|---|---|---|---|
| 1, crashed after 2 | 0 | none | 0 and 1 |
| 2, resume | 2 | 0 and 1 | 2 and 3 |
| 3, resume again | 4 | 0 to 3 | none |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a resume jumps straight to the step that failed. It runs the whole workflow again from the top, and the engine answers each finished step from the journal. So any code outside a step runs every time: put each side effect inside a step.
Try it: a journal in a dictionary
journal = {} # in the lab this is a table in PostgreSQL
def step(seq, name, work):
if seq in journal:
print("replay", seq, name, journal[seq])
return journal[seq]
result = work()
journal[seq] = result # written down before going on
print("run ", seq, name, result)
return result
def order(crash_after):
step(0, "reserve_stock", lambda: "2 reserved")
step(1, "charge_card", lambda: "2500 charged")
if crash_after == 2:
print("crash")
return
step(2, "send_receipt", lambda: "emailed")
order(crash_after=2) # the first run dies after two steps
order(crash_after=0) # the rerun replays two and runs one
# Change the first call to crash_after=0 and compare.
The code must ask for the same steps
Replay works only if the code asks for the same steps, in the same order, on every run. The lab's engine checks the name of each step against its row and stops if they differ.
So a workflow must not make choices from things that change between runs, such as the clock or a random number. Make such a value a step, and its result is kept like any other.