How to retry without a storm
A rate limit is a rule that a service will only accept so many calls in a short time. A retry is a second try after a call was refused. A careless retry can make an overload worse.
The module on cost, tracing and reliability retries once and then falls back. This lesson goes further. It adds waits that grow, a spread so callers do not move together, a limit on tries and a retry that is safe to repeat.
By the end of this lesson
Note
I can retry a call without causing a storm.
Key words
| Word | What it means | Example |
|---|---|---|
| Rate limit | the most calls a service accepts in a time window | |
| Backoff | a wait that grows after each failed try | 400, 800, 1600 |
| Spread | a small different wait for each caller | |
| Storm | many callers retrying at the same moment | |
| Idempotent | safe to repeat: doing it twice has the effect of once | |
| Simulated clock | a number we move by hand instead of waiting |
The cells students fill in are left blank in this preview.
How a rate limit counts
The service cuts time into windows of 1000 milliseconds. In each window it accepts a fixed number of calls, here 2. It refuses the rest until the next window starts.
Time here is a number that we move on by hand. Nothing waits, so every run gives the same answer.
Worked example: 2 calls per window
| Time (ms) | Window | Calls already accepted | Result |
|---|---|---|---|
| 0 | 0 | 0 | Accepted |
| 0 | 0 | 1 | Accepted |
| 0 | 0 | 2 | Refused, the window is full |
| 400 | 0 | 2 | Refused again |
| 1200 | 1 | 0 | Accepted, a new window |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think retrying at once is the quickest way to get an answer. If the service is overloaded, every instant retry adds load, and the service gets worse for everybody. Wait first, and wait longer each time.
Backoff and spread
Backoff is a wait that doubles. With a base of 400, the waits are 400, 800, 1600. Put a cap on it, here 2000, or the waits grow without end.
If six callers fail together and all wait 400, they all return together. A spread adds a small different number for each caller, so they come back one by one. Real systems pick the spread at random, called jitter. Here it is a fixed sum, so every run repeats.
Worked example: base 400, cap 2000
| Failed attempt | Backoff | Spread for request 3 | Next try at (from 0) |
|---|---|---|---|
| 1 | 400 | 34 | 434 |
| 2 | 800 | 47 | 1281 (from 434) |
| 3 | 1600 | 10 | 2891 (from 1281) |
| 4 | 2000, the cap | Not used | Give up if tries is 4 |
The cells students fill in are left blank in this preview.
A retry must be safe to repeat
Sometimes a call worked but the answer was lost on the way back. The caller thinks it failed and tries again. If the retry charges or sends something again, you have done it twice.
Give every action a key. The server remembers keys it has seen. A repeat with a seen key does nothing new.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the retry script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'function delay(attempt, base, cap) { return Math.min(base * Math.pow(2, attempt - 1), cap); }' 'function spread(id, attempt) { return (id * 7 + attempt * 13) % 50; }' 'function run(count, limit, window, base, cap, tries, spreadOn) {' ' const pending = [];' ' const history = {};' ' const counts = {};' ' for (let id = 1; id <= count; id++) { pending.push([0, id, 1]); history[id] = []; }' > retry.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the retry script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' while (pending.length > 0) {' ' pending.sort(function (a, b) { return a[0] - b[0] || a[1] - b[1] || a[2] - b[2]; });' ' const item = pending.shift();' ' const slot = Math.floor(item[0] / window);' ' history[item[1]].push(item[0]);' ' counts[slot] = counts[slot] || 0;' ' if (counts[slot] < limit) { counts[slot] = counts[slot] + 1; continue; }' ' if (item[2] < tries) {' ' const extra = spreadOn ? spread(item[1], item[2]) : 0;' ' pending.push([item[0] + delay(item[2], base, cap) + extra, item[1], item[2] + 1]);' ' }' ' }' ' return history;' '}' >> retry.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the retry script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' for (const on of [false, true]) {' ' const h = run(6, 2, 1000, 400, 2000, 4, on);' ' const seen = {};' ' let worst = 0;' ' for (const id in h) h[id].slice(1).forEach(function (t) { seen[t] = (seen[t] || 0) + 1; worst = Math.max(worst, seen[t]); });' ' console.log((on ? \"with spread\" : \"no spread\") + \": most retries in one millisecond \" + worst);' ' console.log(\" request 3 tried at \" + h[3].join(\" \") + \", request 5 at \" + h[5].join(\" \"));' ' }' ' const reply = await ask([{ role: \"user\", content: \"In one short sentence, why add a random wait to a retry?\" }], \"So many callers do not all retry at the same moment and overload the service again.\", 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> retry.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the retries
docker exec -w /home/node/project agent node retry.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| no spread: most retries in one millisecond 4 | Four callers came back at the same moment |
| request 3 tried at 0 400 1200, request 5 at 0 400 1200 2800 | Without a spread the times are tidy and the same |
| with spread: most retries in one millisecond 1 | No two retries share a moment |
| request 3 tried at 0 434 1281 2891, request 5 at 0 448 1259 2883 | Each caller waits a little differently |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.