Which durable execution engine should you use?
Temporal, Restate, DBOS and AWS Lambda durable functions are durable execution engines: they do what the lab's hundred lines do, for many workflows at once. Each keeps a journal and replays it.
They differ in where the journal lives and how your program reaches it. Sometimes the right answer is none of them.
By the end of this lesson
Note
I can choose between a durable execution engine and a simpler tool for a job.
Key words
| Word | What it means |
|---|---|
| Engine | software that runs workflows and keeps their journals |
| Library | code you import into your own program |
| Server | a separate program your code talks to |
| Activity | Temporal's name for a step with side effects |
| Deterministic | making the same choices every time it runs |
The cells students fill in are left blank in this preview.
Every engine keeps a journal and replays it
The lab's engine is two files: a client for PostgreSQL and the engine itself. It has the whole idea: a step runs once, its result is written down, and a rerun replays it.
A real engine adds what the lab leaves out. Many workers share the work, queues hand it out and timers wake on their own. There are tools for changing code and a web page to watch it all.
Count the lines of the lab's engine
docker exec worker wc -l pg.py engine.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: where each keeps the journal
| Engine | Where the journal lives | How your code reaches it |
|---|---|---|
| Temporal | the Temporal service's own database | your workers ask a task queue for work |
| Restate | the Restate server's log | the server calls your handlers |
| DBOS | your own PostgreSQL | a library inside your program |
| AWS Lambda durable functions | kept by AWS for the function | steps and waits in the function's code |
| This lab | a journal table in PostgreSQL | a library, engine.py |
The cells students fill in are left blank in this preview.
Workflow code must be deterministic
Temporal splits a program in two. Workflow code decides what happens next and must be deterministic. Activities do the work with side effects, such as charging a card.
Replay is why. A resume runs the workflow code again and matches each step to the journal. The next Runs crash an order, then resume it with a changed version of the code that sends the receipt first.
Crash an order with version 1
docker exec worker python3 order.py order-10 --fresh --crash-after 2Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Resume it with version 2
docker exec worker python3 order.py order-10 --v2Shown here, not run. Press Start lab to type it into your own lab’s terminal.
A common mistake
Warning
Many people change a workflow's code while runs are part way through. Those runs replay their journals against the new code, and a changed order of steps no longer matches. Change code only with a version check, or let old runs finish first.
Try it: which change is safe?
journal = ["reserve_stock", "charge_card"] # written by version 1
version_1 = ["reserve_stock", "charge_card", "send_receipt", "book_courier"]
version_2 = ["reserve_stock", "send_receipt", "charge_card", "book_courier"]
def check(code):
for seq in range(len(journal)):
if journal[seq] != code[seq]:
return "stops at step " + str(seq)
return "replays, carries on at step " + str(len(journal))
print("version 1:", check(version_1))
print("version 2:", check(version_2))
# Make version_2 add a new step at the end instead, and run it again.
When you do not need an engine
An engine is another thing to run, learn and keep in step with your code. A job of one step that is safe to repeat needs a plain retry.
A job whose effects are all idempotent can sit in a queue that delivers at least once. Reach for an engine when a job has several steps with effects, waits hours or days, or must undo what it did.
Worked example: three jobs, three tools
| Job | What it needs | A good fit |
|---|---|---|
| Resize one uploaded photo | one step, safe to repeat | a plain retry |
| Send each new order to a warehouse, keyed by order | repeats are harmless | a queue with retries |
| Take an order, charge, wait for stock for days, refund on failure | steps, long waits, undo | a durable engine |
The cells students fill in are left blank in this preview.