Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Choosing a durable execution engine

After this lesson: I can choose between a durable execution engine and a simpler tool for a job.

60 min lesson3 commandsDurable execution lab: crash a workflow on purpose and resume it from its journal

Which durable execution engine should you use?

Temporal, Restate, DBOS and AWS Lambda durable functions are durable execution engines: they do what the lab's hundred lines do, for many workflows at once. Each keeps a journal and replays it.

They differ in where the journal lives and how your program reaches it. Sometimes the right answer is none of them.

By the end of this lesson

Note

I can choose between a durable execution engine and a simpler tool for a job.

Key words

WordWhat it means
Enginesoftware that runs workflows and keeps their journals
Librarycode you import into your own program
Servera separate program your code talks to
ActivityTemporal's name for a step with side effects
Deterministicmaking the same choices every time it runs

The cells students fill in are left blank in this preview.

Every engine keeps a journal and replays it

The lab's engine is two files: a client for PostgreSQL and the engine itself. It has the whole idea: a step runs once, its result is written down, and a rerun replays it.

A real engine adds what the lab leaves out. Many workers share the work, queues hand it out and timers wake on their own. There are tools for changing code and a web page to watch it all.

Count the lines of the lab's engine

Command
docker exec worker wc -l pg.py engine.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: where each keeps the journal

EngineWhere the journal livesHow your code reaches it
Temporalthe Temporal service's own databaseyour workers ask a task queue for work
Restatethe Restate server's logthe server calls your handlers
DBOSyour own PostgreSQLa library inside your program
AWS Lambda durable functionskept by AWS for the functionsteps and waits in the function's code
This laba journal table in PostgreSQLa library, engine.py

The cells students fill in are left blank in this preview.

Workflow code must be deterministic

Temporal splits a program in two. Workflow code decides what happens next and must be deterministic. Activities do the work with side effects, such as charging a card.

Replay is why. A resume runs the workflow code again and matches each step to the journal. The next Runs crash an order, then resume it with a changed version of the code that sends the receipt first.

Crash an order with version 1

Command
docker exec worker python3 order.py order-10 --fresh --crash-after 2

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Resume it with version 2

Command
docker exec worker python3 order.py order-10 --v2

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A common mistake

Warning

Many people change a workflow's code while runs are part way through. Those runs replay their journals against the new code, and a changed order of steps no longer matches. Change code only with a version check, or let old runs finish first.

Try it: which change is safe?

Code: python
journal = ["reserve_stock", "charge_card"]   # written by version 1

version_1 = ["reserve_stock", "charge_card", "send_receipt", "book_courier"]
version_2 = ["reserve_stock", "send_receipt", "charge_card", "book_courier"]

def check(code):
    for seq in range(len(journal)):
        if journal[seq] != code[seq]:
            return "stops at step " + str(seq)
    return "replays, carries on at step " + str(len(journal))

print("version 1:", check(version_1))
print("version 2:", check(version_2))
# Make version_2 add a new step at the end instead, and run it again.

When you do not need an engine

An engine is another thing to run, learn and keep in step with your code. A job of one step that is safe to repeat needs a plain retry.

A job whose effects are all idempotent can sit in a queue that delivers at least once. Reach for an engine when a job has several steps with effects, waits hours or days, or must undo what it did.

Worked example: three jobs, three tools

JobWhat it needsA good fit
Resize one uploaded photoone step, safe to repeata plain retry
Send each new order to a warehouse, keyed by orderrepeats are harmlessa queue with retries
Take an order, charge, wait for stock for days, refund on failuresteps, long waits, undoa durable engine

The cells students fill in are left blank in this preview.

About this lab and its source: Durable execution lab: crash a workflow on purpose and resume it from its journal

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.