Some errors fix themselves
A cloud service answers every call, and sometimes the answer is an error. An error code tells you which kind it is.
A missing S3 bucket stays missing if you ask again. A busy DynamoDB table may answer a second later. Telling the two apart, and retrying only the right one, saves time and stops you making a busy service busier.
By the end of this lesson
Note
I can tell an error to retry from one to fix.
Key words
| Word | What it means | Example |
|---|---|---|
| Error code | a short name for what went wrong | NoSuchKey |
| Status code | a number that sorts an answer into a family | 404 |
| Throttling | a service refusing calls because you sent too many | |
| Retry | to send the same call again | |
| Backoff | waiting longer before each retry | |
| Jitter | a random share of the wait, so callers do not move together |
The cells students fill in are left blank in this preview.
An error has a code and a status
When a call fails, boto3 gives you an error. It holds a status code, such as 404, and an error code, such as NoSuchKey.
Test the error code and the status. Do not test the message: its wording can change between services, and the emulator wording differs from the real one in places.
Worked example: errors read from captured answers
| Call | Code | Status | Retry? |
|---|---|---|---|
| get_object, key missing | NoSuchKey | 404 | No, fix the key |
| head_object, key missing | 404 | 404 | No, fix the key |
| list_objects_v2, bucket missing | NoSuchBucket | 404 | No, make the bucket |
| put_item, condition false | ConditionalCheckFailedException | 400 | No, the answer is the answer |
| query, wrong key name | ValidationException | 400 | No, fix the call |
| send_message, queue missing | AWS.SimpleQueueService.NonExistentQueue | 400 | No, make the queue |
The cells students fill in are left blank in this preview.
Run: read the code and the status
answers = [
{"error": {"Code": "NoSuchKey", "Message": "Missing.", "HTTPStatusCode": 404}},
{"error": {"Code": "ConditionalCheckFailedException", "Message": "No.", "HTTPStatusCode": 400}},
{"error": {"Code": "InternalServerError", "Message": "Oops.", "HTTPStatusCode": 500}},
{"Items": []},
]
for answer in answers:
if "error" not in answer:
print("worked")
else:
error = answer["error"]
print(error["Code"], error["HTTPStatusCode"])
Retry what may work next time
Some errors say "not now". A throttling error means you sent too many calls. A status of 500 or above means the service had a fault. Both may pass if you wait.
Other errors say "not this call". Your call was wrong, so the same call fails every time. Fix it instead.
A common mistake
Warning
Many people retry every error to be safe. A wrong key then fails five times and wastes the wait. Retry only throttling and server faults. Fix every other error, because the same call gets the same answer.
Run: sort errors into retry and fix
# Throttling codes are the documented ones, not captured here.
THROTTLING = ["ThrottlingException", "ProvisionedThroughputExceededException", "SlowDown"]
def worth_retrying(error):
if error["HTTPStatusCode"] >= 500:
return True
return error["Code"] in THROTTLING
errors = [
{"Code": "NoSuchKey", "HTTPStatusCode": 404},
{"Code": "SlowDown", "HTTPStatusCode": 503},
{"Code": "ThrottlingException", "HTTPStatusCode": 400},
{"Code": "ValidationException", "HTTPStatusCode": 400},
]
for error in errors:
print(error["Code"], worth_retrying(error))
# Add "RequestLimitExceeded" to THROTTLING and test it.
Backoff waits longer each time
If a service is busy, a retry a millisecond later makes it busier. So wait, and wait longer after each failure. Doubling the wait is common. Set a cap so the wait never grows past a limit.
Now add jitter: wait a random share of that time. Without it, a hundred callers all wake together and fail together again.
Worked example: the ceiling of each wait
| Attempt | Doubled | With a cap of 8 | Wait if the pick is 0.5 |
|---|---|---|---|
| 1 | 0.5 seconds | 0.5 | 0.25 |
| 2 | 1 second | 1 | 0.5 |
| 3 | 2 seconds | 2 | 1 |
| 4 | 4 seconds | 4 | 2 |
| 5 | 8 seconds | 8 | 4 |
| 6 | 16 seconds | 8 | 4 |
The cells students fill in are left blank in this preview.
Run: a wait you can test
# pick is a number from 0 to 1. A real caller passes random.random().
# A test passes a fixed number, so the answer never changes.
def backoff(attempt, base, cap, pick):
ceiling = min(cap, base * 2 ** (attempt - 1))
return ceiling * pick
for attempt in range(1, 7):
print(attempt, backoff(attempt, 0.5, 8, 1.0), backoff(attempt, 0.5, 8, 0.5))
# Change the cap from 8 to 2. Which attempts change?