Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Idempotency keys and exactly-once effects

After this lesson: I can make a repeated step safe with an idempotency key.

60 min lesson6 commandsDurable execution lab: crash a workflow on purpose and resume it from its journal

What is an idempotency key?

An idempotency key is a unique name sent with a request, so that the receiver acts on it once however many times it arrives. It closes the last gap in a step journal: a crash after an effect but before its row.

Card providers, email services and message queues accept such keys for exactly this reason. With a journal and a key together, an effect happens once.

By the end of this lesson

Note

I can make a repeated step safe with an idempotency key.

Key words

WordWhat it means
Idempotenthaving the same effect whether done once or many times
Idempotency keya name for one request, so repeats are recognised
At least oncea promise that something happens, maybe more than once
Exactly oncean effect that happens one time, never zero or two
Unique constrainta database rule refusing a second row with the same value

The cells students fill in are left blank in this preview.

A crash can land between effect and row

The engine runs a step's work, then writes its journal row. Between those two moments there is a gap. If the process dies in the gap, the effect has happened but the journal does not know.

On the resume there is no row for that step, so the engine runs it again. The journal gives at least once, not exactly once. The lab's --crash-inside flag crashes in exactly that gap.

Crash inside the charge, with no key

Command
docker exec worker python3 order.py order-8 --fresh --crash-inside charge_card

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Resume order-8

Command
docker exec worker python3 order.py order-8

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Count the payments for order-8

Command
docker exec db psql -U durable -c "SELECT workflow, amount FROM payments WHERE workflow = 'order-8'"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

The key turns a repeat into an answer

With --keyed, the charge carries the key order-9:charge_card. The lab's provider keeps payments in a table where the key is unique, and inserts with ON CONFLICT DO NOTHING.

The first charge adds a row. The repeat finds the key already there, adds nothing and answers with "repeat": true. The workflow carries on with the same result either way.

Crash inside the charge, with a key

Command
docker exec worker python3 order.py order-9 --fresh --keyed --crash-inside charge_card

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Resume order-9 with its key

Command
docker exec worker python3 order.py order-9 --keyed

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Count the payments for order-9

Command
docker exec db psql -U durable -c "SELECT workflow, idem_key, amount FROM payments WHERE workflow = 'order-9'"

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: the same crash, two ways

OrderKey sentWhat the resume gotPayments
order-8nonea second charge2
order-9order-9:charge_cardan answer marked repeat1

The cells students fill in are left blank in this preview.

A common mistake

Warning

Many people make a fresh random key on every call. Then the resume sends a new key, and the provider charges again. A key must be the same on every rerun: build it from the workflow and the step, or keep it in the journal.

Try it: a key that changes

Code: python
import uuid

taken = {}   # the card provider: one payment per key

def charge(key, pence):
    if key in taken:
        return "repeat, nothing taken"
    taken[key] = pence
    return "charged " + str(pence)

def make_key(order):
    return order + ":charge_card"   # the same on every rerun

# The workflow sends the charge, crashes, and sends it again.
for attempt in [1, 2]:
    print("attempt", attempt, charge(make_key("order-3"), 2500))
print("payments taken:", len(taken))
# Make make_key return order + str(uuid.uuid4()) and run it again.

About this lab and its source: Durable execution lab: crash a workflow on purpose and resume it from its journal

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.