What is an idempotency key?
An idempotency key is a unique name sent with a request, so that the receiver acts on it once however many times it arrives. It closes the last gap in a step journal: a crash after an effect but before its row.
Card providers, email services and message queues accept such keys for exactly this reason. With a journal and a key together, an effect happens once.
By the end of this lesson
Note
I can make a repeated step safe with an idempotency key.
Key words
| Word | What it means |
|---|---|
| Idempotent | having the same effect whether done once or many times |
| Idempotency key | a name for one request, so repeats are recognised |
| At least once | a promise that something happens, maybe more than once |
| Exactly once | an effect that happens one time, never zero or two |
| Unique constraint | a database rule refusing a second row with the same value |
The cells students fill in are left blank in this preview.
A crash can land between effect and row
The engine runs a step's work, then writes its journal row. Between those two moments there is a gap. If the process dies in the gap, the effect has happened but the journal does not know.
On the resume there is no row for that step, so the engine runs it again. The journal gives at least once, not exactly once. The lab's --crash-inside flag crashes in exactly that gap.
Crash inside the charge, with no key
docker exec worker python3 order.py order-8 --fresh --crash-inside charge_cardShown here, not run. Press Start lab to type it into your own lab’s terminal.
Resume order-8
docker exec worker python3 order.py order-8Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Count the payments for order-8
docker exec db psql -U durable -c "SELECT workflow, amount FROM payments WHERE workflow = 'order-8'"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
The key turns a repeat into an answer
With --keyed, the charge carries the key order-9:charge_card. The lab's provider keeps payments in a table where the key is unique, and inserts with ON CONFLICT DO NOTHING.
The first charge adds a row. The repeat finds the key already there, adds nothing and answers with "repeat": true. The workflow carries on with the same result either way.
Crash inside the charge, with a key
docker exec worker python3 order.py order-9 --fresh --keyed --crash-inside charge_cardShown here, not run. Press Start lab to type it into your own lab’s terminal.
Resume order-9 with its key
docker exec worker python3 order.py order-9 --keyedShown here, not run. Press Start lab to type it into your own lab’s terminal.
Count the payments for order-9
docker exec db psql -U durable -c "SELECT workflow, idem_key, amount FROM payments WHERE workflow = 'order-9'"Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: the same crash, two ways
| Order | Key sent | What the resume got | Payments |
|---|---|---|---|
| order-8 | none | a second charge | 2 |
| order-9 | order-9:charge_card | an answer marked repeat | 1 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people make a fresh random key on every call. Then the resume sends a new key, and the provider charges again. A key must be the same on every rerun: build it from the workflow and the step, or keep it in the journal.
Try it: a key that changes
import uuid
taken = {} # the card provider: one payment per key
def charge(key, pence):
if key in taken:
return "repeat, nothing taken"
taken[key] = pence
return "charged " + str(pence)
def make_key(order):
return order + ":charge_card" # the same on every rerun
# The workflow sends the charge, crashes, and sends it again.
for attempt in [1, 2]:
print("attempt", attempt, charge(make_key("order-3"), 2500))
print("payments taken:", len(taken))
# Make make_key return order + str(uuid.uuid4()) and run it again.