How long should a job wait to retry?
A retry runs a failed step again, and backoff is the wait before each retry, growing every time. A durable timer is a wait written in the journal, so that a crash does not start it again.
Together they let a workflow ride out a slow card provider, or wait three days before it sends a reminder for an unpaid invoice.
By the end of this lesson
Note
I can work out a retry schedule with backoff, a cap and a limit.
Key words
| Word | What it means |
|---|---|
| Timeout | the longest you wait for one answer |
| Exponential backoff | waits that double after each failure |
| Cap | the longest a single wait may be |
| Jitter | a random part of each wait |
| Durable timer | a wait whose wake-up time is kept in the journal |
The cells students fill in are left blank in this preview.
A timeout turns a hang into a failure
A service that never answers is worse than one that refuses: the step would wait for ever. So each attempt has a timeout, 2 seconds in the lab, after which it counts as failed.
A failed attempt is retried after a wait. Waiting longer each time gives a busy service room to recover. In the lab the first wait is 0.5 seconds and each one doubles.
Retry a provider that times out three times
docker exec worker python3 retry.py --fail 3Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: the waits the lab printed
| Attempt | Result | Wait before the next | How it is worked out |
|---|---|---|---|
| 1 | timed out | 0.5 s | 0.5 times 1 |
| 2 | timed out | 1.0 s | 0.5 times 2 |
| 3 | timed out | 2.0 s | 0.5 times 4 |
| 4 | charged | none | stop: it worked |
The cells students fill in are left blank in this preview.
A cap, jitter and a limit
Doubling for ever soon means waiting hours, so the lab caps each wait at 4 seconds. When a thousand workflows fail together, equal waits make them all retry together too. Jitter picks each wait at random between 0 and the backoff, which spreads them out.
Finally there is a limit on attempts. After the last one the step fails, and the workflow decides what happens next.
Six attempts with jitter, then give up
docker exec worker python3 retry.py --fail 9 --attempts 6 --jitterShown here, not run. Press Start lab to type it into your own lab’s terminal.
A common mistake
Warning
Many people retry every error. Retry only an error that may pass, such as a timeout or a busy server. A declined card or a wrong address fails the same way every time, so that step should fail at once.
Try it: change the backoff
BASE = 0.5 # seconds before the first retry
FACTOR = 2 # each wait is twice the one before
CAP = 4.0 # no single wait is longer than this
ATTEMPTS = 7
total = 0
for retry in range(ATTEMPTS - 1):
wait = min(CAP, BASE * FACTOR ** retry)
total = total + wait
print("before retry", retry + 1, "wait", wait, "s")
print("waiting in all:", total, "s")
# Change CAP to 1.0, then FACTOR to 3, and run it again.
A durable timer is a row
Some waits are long: send an invoice, then a reminder three days later. A sleeping process for three days would be lost at the first restart.
A durable timer writes its wake-up time as a journal step, then the process may stop. Each later run replays that row and compares it with the clock. The lab's clock is the number given to --now, in seconds.
Send the invoice and set the timer
docker exec worker python3 remind.py inv-1 --fresh --now 0Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Run it again one day later
docker exec worker python3 remind.py inv-1 --now 86400Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Run it after three days
docker exec worker python3 remind.py inv-1 --now 260000Shown here, not run. Press Start lab to type it into your own lab’s terminal.