Why a payment can arrive twice
At-least-once delivery means a message is never lost, but it may arrive more than once. Networks drop replies, so a sender that does not hear back sends again.
If your consumer adds a payment each time it sees one, a retry takes the money twice. You will learn how to build a consumer that survives retries.
By the end of this lesson
Note
I can design a consumer that survives retries.
Key words
| Word | What it means |
|---|---|
| At-least-once delivery | no message is lost, but some repeat |
| Retry | sending again after no answer |
| Duplicate | a message that arrives a second time |
| Idempotent | doing it twice has the same effect as once |
| Offset | the place number of a message in the log |
| Commit | saving the offset you have finished |
The cells students fill in are left blank in this preview.
A lost reply makes the sender try again
A producer sends a message and waits for an acknowledgement, a short reply that says it arrived. If the reply is lost, the producer cannot tell whether the message arrived. The safe choice is to send it again.
That is a retry. The log now holds the message twice. Both copies are real messages as far as the broker can tell.
Three delivery guarantees
| Name | What the sender does | What can go wrong |
|---|---|---|
| At most once | Never retries | A message can be lost |
| At least once | Retries until acknowledged | A message can repeat |
| Exactly once | Retries, and the receiver ignores repeats | Needs extra work in the receiver |
The cells students fill in are left blank in this preview.
Make a topic of one partition
curl -s -X POST -H "Content-Type: application/json" -d '{"name":"payments","partitions":1,"replicationFactor":1,"configs":{}}' http://kafka-ui:8080/api/clusters/lab/topics -o /dev/null -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Send payment p17
curl -s -X POST -H "Content-Type: application/json" -d '{"partition":0,"key":"p17","value":"paidada","headers":{},"keySerde":"String","valueSerde":"String"}' http://kafka-ui:8080/api/clusters/lab/topics/payments/messages -o /dev/null -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Send p17 again, as a retry would
curl -s -X POST -H "Content-Type: application/json" -d '{"partition":0,"key":"p17","value":"paidada","headers":{},"keySerde":"String","valueSerde":"String"}' http://kafka-ui:8080/api/clusters/lab/topics/payments/messages -o /dev/null -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Send a different payment, p18
curl -s -X POST -H "Content-Type: application/json" -d '{"partition":0,"key":"p18","value":"paidbo","headers":{},"keySerde":"String","valueSerde":"String"}' http://kafka-ui:8080/api/clusters/lab/topics/payments/messages -o /dev/null -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Look at the offset of the next message
curl -s http://kafka-ui:8080/api/clusters/lab/topics/payments | grep -o '"partition":[0-9]*\|"offsetMax":[0-9]*'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Read the whole topic
curl -s -N "http://kafka-ui:8080/api/clusters/lab/topics/payments/messages/v2?seekType=BEGINNING&limit=10" | grep -o '"partition":[0-9]*,"offset":[0-9]*\|"value":"[a-z]*"'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A crash before a commit repeats work
A consumer reads messages in order and saves its place by committing an offset. After a restart it carries on from the last committed offset.
If it handles a message and crashes before it commits, the restart reads that message again. Committing before handling avoids the repeat, but a crash then loses the message. Most systems choose the repeat.
Worked example: a crash between handling and commit
| Step | Offset handled | Committed offset | Balance |
|---|---|---|---|
| Handle offset 0 (amount 5) | 0 | 0 | 5 |
| Handle offset 1 (amount 3) | 1 | 0 | 8 |
| Crash, then restart at 0 | 0 again | 0 | 13 |
| Handle offset 1 again | 1 again | 0 | 16 |
The cells students fill in are left blank in this preview.
Watch a restart add the payments twice
# A consumer adds up payments. It crashes before it saves its place.
log = [5, 3, 4]
balance = 0
committed = 0
for offset in range(committed, 2): # reads offsets 0 and 1, then crashes
balance = balance + log[offset]
print("Before the crash: balance", balance, "committed", committed)
for offset in range(committed, len(log)): # the restart begins at the saved place
balance = balance + log[offset]
print("After the restart: balance", balance, "but the true total is 12")
# Add committed = 2 above the restart loop and run again.
A common mistake
Warning
Many people think the log removes a repeated message. A plain log keeps every message that is written, as the lab shows. Your consumer must spot a repeat itself, by an id inside the message, not by its offset.
An idempotent consumer skips repeated messages
An action is idempotent when doing it twice has the same effect as doing it once. Setting a value to 7 is idempotent. Adding 5 is not.
To make adding safe, give each message an id and keep a record of the ids you have handled. If an id is already there, skip the message. Save the record and the result in the same step, so a crash cannot split them.
Skip an id that was already handled
# Each payment has an id. Remember the ids you have handled.
seen = {}
total = 0
for message in [["p17", 5], ["p17", 5], ["p18", 3]]:
if message[0] in seen:
print(message[0], "already handled, skipped")
else:
seen[message[0]] = True
total = total + message[1]
print(message[0], "added, total", total)
# Change the second p17 to p19 and run again.