Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

A step journal and replay

After this lesson: I can use a step journal to find where a crashed workflow should carry on.

60 min lesson4 commandsDurable execution lab: crash a workflow on purpose and resume it from its journal

How does a workflow survive a crash?

Durable execution writes each finished step of a workflow in a journal, so that a rerun can carry on after a crash. The journal is a table in a database, which outlives the process.

On a rerun, a step that has a row is replayed from it, not run again. Work starts at the first step with no row. The lab's engine does this in about a hundred lines.

By the end of this lesson

Note

I can use a step journal to find where a crashed workflow should carry on.

Key words

WordWhat it means
Workflowa job whose steps must all happen, in order
Journala table holding each finished step and its result
Sequence numbera step's place in the workflow, counted from 0
Replaytaking a step's result from the journal instead of running it
Resumerunning a crashed workflow again so it carries on

The cells students fill in are left blank in this preview.

The engine writes each result before going on

In the lab, a workflow is ordinary Python. Each step is a call to wf.step with a name and a function to run. The engine gives every step a sequence number: 0 for the first, 1 for the next.

When a step finishes, the engine inserts a row into the journal table: the workflow, the number, the name and the result. Only then does the workflow go on.

Crash the order workflow after two steps

Command
docker exec worker python3 order.py order-7 --fresh --crash-after 2

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Read the journal rows for order-7

Command
docker exec db psql -U durable -c "SELECT seq, step, result FROM journal WHERE workflow = 'order-7' ORDER BY seq"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A resume replays the rows it finds

Now run the same workflow with no flags. The engine reads the journal first and finds two rows.

Steps 0 and 1 print replay: their results come from the rows, so nothing is reserved or charged again. Steps 2 and 3 have no rows, so they print run. Running it once more replays all four steps and does nothing new.

Resume order-7

Command
docker exec worker python3 order.py order-7

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Resume order-7 once more

Command
docker exec worker python3 order.py order-7

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: three runs of order-7

RunRows foundReplayedRan
1, crashed after 20none0 and 1
2, resume20 and 12 and 3
3, resume again40 to 3none

The cells students fill in are left blank in this preview.

A common mistake

Warning

Many people think a resume jumps straight to the step that failed. It runs the whole workflow again from the top, and the engine answers each finished step from the journal. So any code outside a step runs every time: put each side effect inside a step.

Try it: a journal in a dictionary

Code: python
journal = {}   # in the lab this is a table in PostgreSQL

def step(seq, name, work):
    if seq in journal:
        print("replay", seq, name, journal[seq])
        return journal[seq]
    result = work()
    journal[seq] = result   # written down before going on
    print("run   ", seq, name, result)
    return result

def order(crash_after):
    step(0, "reserve_stock", lambda: "2 reserved")
    step(1, "charge_card", lambda: "2500 charged")
    if crash_after == 2:
        print("crash")
        return
    step(2, "send_receipt", lambda: "emailed")

order(crash_after=2)   # the first run dies after two steps
order(crash_after=0)   # the rerun replays two and runs one
# Change the first call to crash_after=0 and compare.

The code must ask for the same steps

Replay works only if the code asks for the same steps, in the same order, on every run. The lab's engine checks the name of each step against its row and stops if they differ.

So a workflow must not make choices from things that change between runs, such as the clock or a random number. Make such a value a step, and its result is kept like any other.

About this lab and its source: Durable execution lab: crash a workflow on purpose and resume it from its journal

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.