Durable execution lab: crash a workflow on purpose and resume it from its journal

This lab teaches durable execution, the idea behind workflow engines such as Temporal, Restate and DBOS, from first principles. Each student has a PostgreSQL database and a Python worker holding an engine of about a hundred lines: a step runs once, its result is written to a journal table, and a rerun after a crash replays the rows it finds and carries on.

Students crash workflows on purpose and resume them: a plain job that charges a card twice, an order that replays its journal, a charge made safe by an idempotency key, a retry schedule, a durable timer and a trip booked as a saga. It goes with the Durable execution module. No commercial engine runs in it; the module compares them in words.

What students learn

Why a job that dies half way repeats its side effects when it is retried from the top; how a journal of finished steps lets a rerun carry on instead; why a crash between an effect and its journal row still repeats the effect, and how an idempotency key closes that gap; how retries back off, with a cap, jitter and a limit; how a durable timer waits as a row rather than a running process; and how a saga undoes finished steps, newest first, when a later one fails. They read the journal with SQL, and see why workflow code must ask for the same steps on every run. It suits a distributed systems or software engineering course, or a module on back-end reliability.

Run it in your browser

The lab starts two services: db, PostgreSQL, and worker, Python with the lab’s files in /tmp. The worker starts once the database answers. The terminal beside the lab has no Python of its own, so it reaches them with docker exec. Every Run prints the same every time: nothing reads the clock and the one random choice has a fixed seed.

  1. Open the Container page. Open Build on the bar and choose Run a container. Anybody may open it, and the Git lab and the SQL lab start with no account. Every other lab needs a free account: create one, sign in, then press Start.
  2. Find the lab. The Sample labs card lists every sample. Find Durable execution.
  3. Press Start. Start runs it as it is, in one press. Edit the file first opens the compose file so you can change it before you start.
  4. Open it. On the lab’s page, press Open terminal. There is no link under In your browser because the lab has no web page. Type docker exec worker python3 order.py order-7 --fresh --crash-after 2: it runs two steps and stops with a crash line. Then docker exec worker python3 order.py order-7 replays those two steps from the journal and runs the other two.

What to ask students to hand in

A workflow of their own written with engine.py, for example a sign-up that creates an account, sends a welcome email and books a call, with the output of a run they crashed with --crash-after, the output of the run that resumed it, and its journal rows from psql. Then one sentence on which steps ran twice and which did not. A good answer puts every side effect inside a step and gives anything that takes money an idempotency key.

A basic lesson on this lab

Durable Execution Basics is a small lesson worked in this lab. In it a student crashes a plain job and counts two payments, crashes the order workflow and reads its journal, resumes it, charges a card once with an idempotency key after a crash inside the charge, and books a trip whose car fails so the saga cancels the hotel and the flight.

Free personal accounts have it in the Basics module, which is on Home, every college can copy it from the Library’s Basics folder, and a new college opens with it in a module called Data Labs (Sample). It is deliberately small: it shows a teacher what a lesson worked in a lab is, and the lab does the rest. Copy it into a module of your own to change it.

The code on GitHub

The compose file has no Dockerfile: it runs two stock public images, postgres and python, each pinned to an exact version, with 256 MB and 128 MB of memory. PostgreSQL is under the PostgreSQL License and Python under the PSF License; neither is distributed by us. The worker runs as an unprivileged account. Its files are written by the compose file itself, including pg.py, a small client of PostgreSQL’s own protocol, because no database driver is in a stock Python image. The database asks for no password, since nothing outside the lab can reach it, and the lab has no way out to the internet. Nothing is kept when the lab stops.

Set it as work

Say in the task what to hand in.

  1. Make the lab. In your module add a task and choose the Container lab card, then In a GitHub repository. Give https://github.com/mitsurutech/lab-durable-lab with the branch main. The compose file is the default, compose.yaml. Fork the repository first if you want to change it.
  2. Publish and try it. Publish pins the commit you were shown, so a later change in the repository does not alter a lab you have already set. Start your own copy first.
  3. Hand it out. Assign it like any task, with a due date if you want one, and say in the task what to do and what to hand in.
  4. Mark what is handed in. Open a student’s copy and the work they handed in, mark it, and send it back with a reason if it needs more.

Try this lab, and what it costs

A free personal account can work through the basic lessons in this lab, with enough free credit each month to follow most lessons and no card. For a class, a school or college has its first 365 days free for up to 50 students, then pays per student a year, with every lab included and staff free.

Questions people ask

Is this Temporal, Restate or DBOS?
No. It is an engine of about a hundred lines written for the lab, with the idea those engines share: a journal of finished steps that a rerun replays. Real engines add many workers, queues, tools for changing code and a web page. The Durable execution module compares them in words.
How does a Run crash the workflow?
The flags --crash-after and --crash-inside make the worker stop its own process at a named place, with exit code 3, so the crash is real and the same every time. The next Run, without the flag, resumes from the journal.
Why does a second Run sometimes print different lines?
Because the journal is still there. A resume of a workflow that has finished replays every step and does nothing new. Each Run that starts a workflow carries --fresh, which forgets it first, so it can be pressed again.
What is still there after I stop and start the lab?
Nothing. The journal, the payments and any file a student wrote are forgotten, and the lab writes its own files again when it starts. Copy anything to hand in from the terminal before you stop it.