One order, four systems
A customer pays for an order. The warehouse must pack it, the billing system must record it and the email service must thank the customer.
If the website calls each one, a slow system slows every order. A better way is to write one event to a log and let each system read it when it is ready.
By the end of this lesson
Note
I can explain a log of events and how consumers read it.
Key words
| Word | What it means | Example |
|---|---|---|
| Event | a record that something happened | order paid |
| Topic | a named log of events | |
| Partition | one ordered slice of a topic | |
| Offset | the number of an event in its partition, from zero | |
| Producer | a program that adds events to a topic | |
| Consumer | a program that reads events from a topic |
The cells students fill in are left blank in this preview.
A topic is a log you add to
Kafka stores events in a topic. Think of a ledger: new lines go on the end and old lines are not changed.
A producer adds events. Every event gets an offset, its place number, starting at 0. Events stay for a set time, such as seven days, whether or not anyone has read them.
Worked example: a log with offsets
| Offset | Event | Written by |
|---|---|---|
| 0 | order 1001 paid | website |
| 1 | order 1002 paid | website |
| 2 | order 1001 packed | warehouse |
| 3 | order 1003 paid | website |
The cells students fill in are left blank in this preview.
Make a topic of two partitions
curl -s -X POST -H "Content-Type: application/json" -d '{"name":"shoplog","partitions":2,"replicationFactor":1,"configs":{}}' http://kafka-ui:8080/api/clusters/lab/topics -o /dev/null -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Look at the offsets of the empty topic
curl -s http://kafka-ui:8080/api/clusters/lab/topics/shoplog | grep -o '"partition":[0-9]*\|"offsetMax":[0-9]*'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Partitions share the work
One log on one machine could not take every event of a big shop. So a topic is split into partitions. Each one is its own log with its own offsets.
Events keep their order inside one partition. Between two partitions there is no promised order.
Add an event to partition 0
curl -s -X POST -H "Content-Type: application/json" -d '{"partition":0,"key":"till","value":"pasta","headers":{},"keySerde":"String","valueSerde":"String"}' http://kafka-ui:8080/api/clusters/lab/topics/shoplog/messages -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Add a second event to partition 0
curl -s -X POST -H "Content-Type: application/json" -d '{"partition":0,"key":"till","value":"bread","headers":{},"keySerde":"String","valueSerde":"String"}' http://kafka-ui:8080/api/clusters/lab/topics/shoplog/messages -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Add an event to partition 1
curl -s -X POST -H "Content-Type: application/json" -d '{"partition":1,"key":"web","value":"milk","headers":{},"keySerde":"String","valueSerde":"String"}' http://kafka-ui:8080/api/clusters/lab/topics/shoplog/messages -w "%{http_code}\n"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Look at the offsets again
curl -s http://kafka-ui:8080/api/clusters/lab/topics/shoplog | grep -o '"partition":[0-9]*\|"offsetMax":[0-9]*'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A consumer remembers its own offset
A consumer reads events from a partition, in order, and remembers the offset of the next one it needs. If it stops and starts, it carries on from there.
Two services can read one topic. Each keeps its own offset, so a slow one never holds up a fast one. The gap between the end of the log and a consumer's offset is its lag.
A consumer reads and moves on
# One partition of a topic, as a list. A consumer remembers one number.
partition = ["pasta", "bread", "milk", "rice"]
offset = 2
print("Read:", partition[offset], "at offset", offset)
offset = offset + 1
print("Next offset:", offset)
print("Still to read:", len(partition) - offset)
# Change the first offset to 0 to read from the beginning.
Two consumers, two offsets
# Two services read the same log. Each has its own offset.
log = ["pasta", "bread", "milk"]
offsets = {"billing": 3, "emails": 1}
for name in offsets:
left = len(log) - offsets[name]
print(name, "has", left, "messages still to read")
# Set the emails offset to 3 and run again.
Read the topic from the beginning
curl -s -N "http://kafka-ui:8080/api/clusters/lab/topics/shoplog/messages/v2?seekType=BEGINNING&limit=10" | grep -o '"partition":[0-9]*,"offset":[0-9]*\|"value":"[a-z]*"'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a read removes the event, as when you take a letter from a mailbox. In a log it does not. Reading only moves the consumer's offset. The event stays, so another service can read it too.
Going further: predict the offsets
In the terminal, press the up arrow until the Run for partition 1 comes back. Change milk to eggs and send it twice.
Before you look, write down the offsetMax you expect for each partition. Then press the Run that shows the offsets and compare.