Queues: send, receive and delete: Free Lesson

A queue of orders, like a café queue

A message queue, such as Amazon SQS, holds messages in line until a program is ready to handle them. It lets a busy shop take orders faster than its workers can process them, and lose none.

Think of a café queue. A customer joins at the back, and the server takes the person at the front. A shop drops each new order into an SQS queue, and a worker takes them one at a time.

By the end of this lesson

Note

I can say what happens to a message from send to delete.

Key words

Word	What it means
Queue	a line of messages waiting to be handled
Message	one piece of text waiting in a queue
Receive	to ask the queue for messages to work on
Visibility timeout	how long a received message stays hidden
Receipt handle	the ticket that lets you delete a received message
At-least-once	a message may arrive more than once, but it always arrives at least once

The cells students fill in are left blank in this preview.

A receive hands you a message

send_message puts text on the queue. Its answer holds a MessageId. receive_message asks for messages. Its answer holds a list called Messages.

Each message has a Body, which is the text, and a ReceiptHandle. A body is only text. Many shops store JSON in it, and the worker turns it back into a dictionary with json.loads.

Worked example: a receive answer

Field	What it holds	Example
Messages	a list with one dictionary per message	two messages
MessageId	the id the queue gave at send	m-1
Body	the text that was sent	{"order": 1, "item": "tea"}
ReceiptHandle	the ticket for deleting this receive	h-1
ApproximateReceiveCount	how many times it was received, as text	"1"

The cells students fill in are left blank in this preview.

Run it: read a receive answer

Code: python

import json
answer = {
    "Messages": [
        {"MessageId": "m-1", "ReceiptHandle": "h-1",
         "Body": "{\"order\": 1, \"item\": \"tea\"}",
         "Attributes": {"ApproximateReceiveCount": "1"}},
        {"MessageId": "m-2", "ReceiptHandle": "h-2", "Body": "two",
         "Attributes": {"ApproximateReceiveCount": "1"}},
    ]
}
for message in answer["Messages"]:
    print(message["MessageId"], message["Body"])
order = json.loads(answer["Messages"][0]["Body"])
print("item:", order["item"])
# Change the loop to print the ReceiptHandle as well.

A received message is hidden, not gone

Receive does not remove a message. It hides it for the visibility timeout, which is 30 seconds on the captured queue. Other workers cannot receive it while it is hidden.

When the work is done, call delete_message with the ReceiptHandle. Only then is the message gone. If the worker crashes first, the timeout ends and the message becomes visible again for another worker.

Worked example: a queue's attributes

Attribute	Value	Meaning
VisibilityTimeout	"30"	seconds a received message stays hidden
ApproximateNumberOfMessages	"3"	messages waiting to be received
ApproximateNumberOfMessagesNotVisible	"0"	messages hidden after a receive
MessageRetentionPeriod	"345600"	seconds a message is kept, which is four days

The cells students fill in are left blank in this preview.

Run it: hidden, then visible again

Code: python

TIMEOUT = 30
received_at = {"m-1": 100}


def is_visible(message_id, now):
    if message_id not in received_at:
        return True
    return now >= received_at[message_id] + TIMEOUT


for now in [100, 120, 130, 160]:
    print(now, is_visible("m-1", now))
print("m-2 never received:", is_visible("m-2", 100))
# Change TIMEOUT to 60. At which times is m-1 visible now?

A common mistake

Warning

Many people think a receive removes the message. It only hides it. If you never call delete_message, the message comes back after the timeout and is handled again. Delete it once the work is done, never before.

At-least-once means repeats happen

The queue keeps a message until it is deleted, so a message can arrive twice. A worker may finish the work and crash before the delete.

A receive may also return fewer messages than you asked for, or none. An empty answer has no Messages key, so check for the key before you read it. Write the worker so that a repeat does no harm.

Run it: skip a repeat

Code: python

deliveries = ["m-1", "m-2", "m-1", "m-3"]
done = []
for message_id in deliveries:
    if message_id in done:
        print(message_id, "again: skip it")
    else:
        done.append(message_id)
        print(message_id, "first time: do the work")

empty = {}
if "Messages" in empty:
    print("messages:", len(empty["Messages"]))
else:
    print("messages: 0")
# Add print(empty["Messages"]) at the end. What goes wrong?

Try it in the cloud lab

The cloud lab is a practice copy of S3, SQS and DynamoDB. Press Start the lab on the bar, then press Run on each command below. They use the real AWS command line, so the same words work on a real account. After the delete, press Run on the receive again: it prints nothing, because the queue is empty. This step is for practice. The assignment below stays plain Python, so you can do it without the lab.