Some failures pass if you wait
An API error is a reply with a status code that says the call did not work. Some errors pass if you wait a moment and try again. Others never pass, however often you try.
A good program tells the two apart. It retries the first kind with growing waits, and it stops and reports the second kind.
By the end of this lesson
Note
I can tell a failure to retry from a failure to fix.
Key words
| Word | What it means | Example |
|---|---|---|
| Status code | a number in a reply that says how the call went | 429 |
| Rate limit | a cap on how many calls you may make in a minute | |
| Backoff | waiting longer after each failed try | |
| Jitter | a random change to a wait, so clients do not retry together | |
| Timeout | the longest you will wait for a reply |
The cells students fill in are left blank in this preview.
The status code says what to do next
A status in the 200s means the call worked. A status in the 400s usually means something is wrong with your call, so the same call will fail again.
Some statuses mean a short, passing problem: a timeout, a server that is busy or a rate limit. For those, waiting and trying again is the right response.
Worked example: retry or fix
| Status | What it means | What to do |
|---|---|---|
| 400 | The request is wrong | Fix the request |
| 401 | No valid key | Fix the key |
| 403 | Not allowed, or no credit left | Fix the cause |
| 408, 504 | A timeout | Retry |
| 429 | Too many requests | Wait, then retry |
| 500, 502, 503 | The server has a problem | Retry a few times |
The cells students fill in are left blank in this preview.
Sort statuses and read a recorded 429
docker exec -w /home/node/project agent node -e '/* Which failures are worth another try? */ const kind = (status) => { if (status >= 200 && status < 300) return "ok"; return [408, 429, 500, 502, 503, 504].includes(status) ? "retry" : "fix"; }; const notes = { 200: "it worked", 400: "the request is wrong", 401: "no valid key", 403: "not allowed", 404: "no such address", 408: "the server timed out", 429: "too many requests", 500: "the server broke", 503: "the server is not ready" }; for (const status of Object.keys(notes)) { console.log(status + " " + kind(Number(status)).padEnd(5) + " " + notes[status]); } /* A recorded 429, written into this script as data. No call is made. */ const recorded = { status: 429, headers: { "retry-after": "2" }, body: { error: { message: "Too many requests, slow down." } } }; console.log("Recorded 429: " + recorded.body.error.message + " The server asks for " + recorded.headers["retry-after"] + " seconds.");'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Backoff waits longer each time
If you retry at once, you add to the load that caused the failure. Backoff waits a little, then twice as long, then twice that, up to a cap.
Add jitter: change each wait by a random amount, between half and all of it. Then a hundred clients that failed together do not all retry in the same instant. If the reply has a Retry-After header, use it.
Print a backoff schedule
docker exec -w /home/node/project agent node -e '/* Wait longer after each failure, up to a cap, and add jitter. */ const base = 1; const cap = 8; const delays = []; for (let attempt = 0; attempt < 6; attempt++) delays.push(Math.min(cap, base * 2 ** attempt)); console.log("Delays in seconds: " + delays.join(", ")); const jittered = delays.map((one) => one / 2 + Math.random() * one / 2); console.log("With jitter, one possible run: " + jittered.map((one) => one.toFixed(2)).join(", ")); console.log("Each jittered delay is between half of its delay and all of it.");'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think every error should be retried. It should not. A 400 or a 401 will fail the same way every time, and a loop that retries it only wastes time and credit. Retry only the failures that can pass, and only a few times.
A timeout stops you waiting for ever
Some calls never answer. Set a timeout, the longest you will wait, and treat running out of time as a failure worth a retry.
The helper below puts the pieces together. It has a timeout, a few tries and backoff with jitter. It stops at once for a failure to fix, and it prints a recorded sample when nothing works.
Write retry.js, part 1
docker exec -w /home/node/project agent node -e 'require("fs").writeFileSync("retry.js", ["// retry.js, part 1: which statuses to retry, and how long to wait.", "const lab = require(\"./lab.js\");", "const kind = (status) => {", " if (status >= 200 && status < 300) return \"ok\";", " return [408, 429, 500, 502, 503, 504].includes(status) ? \"retry\" : \"fix\";", "};", "const pause = (ms) => new Promise((done) => setTimeout(done, ms));", "", "// Backoff, the Retry-After header if there is one, a cap, and jitter.", "function waitFor(attempt, response) {", " let wait = 150 * 2 ** attempt;", " if (response) wait = Math.max(wait, Number(response.headers.get(\"retry-after\")) * 1000 || 0);", " wait = Math.min(wait, 3000);", " return wait / 2 + Math.random() * wait / 2;", "}"].join("\n") + "\n")'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Add the retry loop
docker exec -w /home/node/project agent node -e 'require("fs").appendFileSync("retry.js", ["", "// Part 2: try again, but never for a failure to fix.", "async function callWithRetry(body, tries) {", " for (let attempt = 1; attempt <= tries; attempt++) {", " let response = null;", " try {", " response = await lab.send(body, 15000);", " if (response.ok) return { ok: true, data: await response.json() };", " console.log(\"attempt \" + attempt + \": HTTP \" + response.status + \", \" + kind(response.status));", " if (kind(response.status) === \"fix\") return { ok: false, why: \"fix the cause, do not retry\" };", " } catch (error) {", " console.log(\"attempt \" + attempt + \": no answer (\" + error.name + \")\");", " }", " if (attempt < tries) await pause(waitFor(attempt, response));", " }", " return { ok: false, why: \"gave up after \" + tries + \" tries\" };", "}", "", "module.exports = { callWithRetry };"].join("\n") + "\n")'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Call the model with retries
docker exec -w /home/node/project agent node -e 'const { callWithRetry } = require("./retry.js"); (async () => { const body = { max_tokens: 20, messages: [{ role: "user", content: "Say hello in one word." }] }; const result = await callWithRetry(body, 3); if (result.ok) console.log("Model answer: " + result.data.choices[0].message.content); else console.log("Recorded sample: Hello. (" + result.why + ")"); })();'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Going further: change the tries
In retry.js, change 150 to 50, so the first wait is about 100 milliseconds, not 300. In the last Run, change the number of tries from 3 to 5. Run it and read what it prints.
Say what you would choose for a button a person is waiting on, and what for a job that runs overnight.