Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Retries, timeouts and durable timers

After this lesson: I can work out a retry schedule with backoff, a cap and a limit.

60 min lesson5 commandsDurable execution lab: crash a workflow on purpose and resume it from its journal

How long should a job wait to retry?

A retry runs a failed step again, and backoff is the wait before each retry, growing every time. A durable timer is a wait written in the journal, so that a crash does not start it again.

Together they let a workflow ride out a slow card provider, or wait three days before it sends a reminder for an unpaid invoice.

By the end of this lesson

Note

I can work out a retry schedule with backoff, a cap and a limit.

Key words

WordWhat it means
Timeoutthe longest you wait for one answer
Exponential backoffwaits that double after each failure
Capthe longest a single wait may be
Jittera random part of each wait
Durable timera wait whose wake-up time is kept in the journal

The cells students fill in are left blank in this preview.

A timeout turns a hang into a failure

A service that never answers is worse than one that refuses: the step would wait for ever. So each attempt has a timeout, 2 seconds in the lab, after which it counts as failed.

A failed attempt is retried after a wait. Waiting longer each time gives a busy service room to recover. In the lab the first wait is 0.5 seconds and each one doubles.

Retry a provider that times out three times

Command
docker exec worker python3 retry.py --fail 3

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: the waits the lab printed

AttemptResultWait before the nextHow it is worked out
1timed out0.5 s0.5 times 1
2timed out1.0 s0.5 times 2
3timed out2.0 s0.5 times 4
4chargednonestop: it worked

The cells students fill in are left blank in this preview.

A cap, jitter and a limit

Doubling for ever soon means waiting hours, so the lab caps each wait at 4 seconds. When a thousand workflows fail together, equal waits make them all retry together too. Jitter picks each wait at random between 0 and the backoff, which spreads them out.

Finally there is a limit on attempts. After the last one the step fails, and the workflow decides what happens next.

Six attempts with jitter, then give up

Command
docker exec worker python3 retry.py --fail 9 --attempts 6 --jitter

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A common mistake

Warning

Many people retry every error. Retry only an error that may pass, such as a timeout or a busy server. A declined card or a wrong address fails the same way every time, so that step should fail at once.

Try it: change the backoff

Code: python
BASE = 0.5     # seconds before the first retry
FACTOR = 2     # each wait is twice the one before
CAP = 4.0      # no single wait is longer than this
ATTEMPTS = 7

total = 0
for retry in range(ATTEMPTS - 1):
    wait = min(CAP, BASE * FACTOR ** retry)
    total = total + wait
    print("before retry", retry + 1, "wait", wait, "s")
print("waiting in all:", total, "s")
# Change CAP to 1.0, then FACTOR to 3, and run it again.

A durable timer is a row

Some waits are long: send an invoice, then a reminder three days later. A sleeping process for three days would be lost at the first restart.

A durable timer writes its wake-up time as a journal step, then the process may stop. Each later run replays that row and compares it with the clock. The lab's clock is the number given to --now, in seconds.

Send the invoice and set the timer

Command
docker exec worker python3 remind.py inv-1 --fresh --now 0

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Run it again one day later

Command
docker exec worker python3 remind.py inv-1 --now 86400

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Run it after three days

Command
docker exec worker python3 remind.py inv-1 --now 260000

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

About this lab and its source: Durable execution lab: crash a workflow on purpose and resume it from its journal

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.