Why a slow model call becomes a job
A job queue lets a request return at once while slow work finishes elsewhere. A model call can take many seconds, and a web request that waits that long times out or ties up a worker. It matters because a retry of a slow call is where a customer gets charged twice.
A user uploads forty pages to summarise. The server answers "accepted, job-17" in a few milliseconds. The page asks for the status until the summary is ready.
By the end of this lesson
Note
I can move a slow call off the request and report its progress.
Key words
| Word | What it means | Example |
|---|---|---|
| Job | a unit of slow work with a state | |
| Queue | jobs waiting for a worker | |
| Worker | code that takes a job and does it | |
| Idempotency key | a client label that makes a repeat harmless | order-1 |
| Backoff | waiting longer after each failed try |
The cells students fill in are left blank in this preview.
A job has four states
A job is queued, then running, then done or failed. A failed try that will be retried goes back to queued. The job keeps its try count and its result.
The client gets a job id at once. It asks for the status by id. It reads the result only when the state is done.
Worked example: job-2 on its own, failing once
| Step | State | Tries | Clock |
|---|---|---|---|
| Submitted | queued | 0 | 0 s |
| First attempt starts | running | 1 | 2 s |
| Upstream timeout | queued | 1 | 7 s |
| Second attempt starts | running | 2 | 9 s |
| Result stored | done | 2 | 9 s |
The cells students fill in are left blank in this preview.
An idempotency key stops a job running twice
Networks repeat requests. A client may press submit twice, or a proxy may resend. Without a guard, each copy becomes a job and the work, and the bill, is doubled.
The client sends an idempotency key. The queue keeps a map from key to job id. A key it has seen returns the existing job and makes no new one.
Retries are bounded and backed off
Set a maximum number of tries. After that the job is failed and a person or a rule decides what next. Wait longer after each failure so a struggling service gets room.
A job that is done is never run again, even if a sweep visits it twice. A done job is a result, not a request.
A common mistake
Warning
Many people think a retry is always safe. It is safe only when the work is idempotent or the queue remembers it. Retry a call that has already been paid for and finished, and you pay again. Check the state before every run.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const jobs = {}; // job id to job' 'const byKey = {}; // idempotency key to job id' 'let clock = 0; // a fake clock, in seconds' 'let next = 1;' 'function submit(key, payload) {' ' if (key in byKey) return byKey[key];' ' const id = \"job-\" + next++;' ' jobs[id] = { id: id, payload: payload, state: \"queued\", tries: 0, result: null };' ' byKey[key] = id;' ' return id;' '}' > queue.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const failFirst = { \"job-2\": 1 }; // job-2 fails once, then works' 'function work(job) {' ' if (job.tries <= (failFirst[job.id] || 0)) throw new Error(\"upstream timeout\");' ' return job.payload.toUpperCase();' '}' 'function runOne(job) {' ' if (job.state === \"done\" || job.state === \"failed\") return;' ' while (job.tries < 3) {' ' job.tries++;' ' job.state = \"running\";' ' clock += 2;' ' try { job.result = work(job); job.state = \"done\"; return; }' ' catch (error) { job.state = \"queued\"; clock += job.tries * 5; }' ' }' ' job.state = \"failed\";' '}' >> queue.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'submit(\"order-1\", \"invoice a\");' 'submit(\"order-2\", \"invoice b\");' 'submit(\"order-1\", \"invoice a\"); // a client retry: same key, same job' 'Object.values(jobs).forEach(runOne);' 'Object.values(jobs).forEach(runOne); // a second sweep runs nothing' 'for (const id in jobs) console.log(id, jobs[id].state, \"tries\", jobs[id].tries, jobs[id].result);' 'console.log(\"jobs:\", Object.keys(jobs).length, \"clock:\", clock + \"s\");' >> queue.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const sample = \"Your two invoices are ready.\";' ' const text = \"Two jobs finished. Write one short status line for the customer.\";' ' console.log(\"Live call:\", await ask([{ role: \"user\", content: text }], sample, 40));' '}' 'main();' >> queue.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the queue
docker exec -w /home/node/project agent node queue.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A stand-in, not a real queue
The queue here is an object in memory and the clock is a number. If the process stops, every job is lost. A real system keeps jobs in a service made for it, and workers run in other processes.
The ideas carry over: states, a try count, a key map and a bound on retries.
One possible output
| Line | What it shows |
|---|---|
| job-1 done tries 1 INVOICE A | A job that worked first time |
| job-2 done tries 2 INVOICE B | A job that failed once, then worked |
| jobs: 2 clock: 11s | Three submits made two jobs |
| Live call | One short status line, or a labelled Recorded sample |
The cells students fill in are left blank in this preview.