Tell it what was wrong
A repair loop asks a model again after a bad answer, and sends the faults back so it can fix them. It matters because a second try that knows the problem often succeeds where a blind retry would not.
A form-filling tool gets a reply with a word the schema does not allow. Rejecting it loses the work. Asking again, with the fault in the message, usually saves it.
By the end of this lesson
Note
I can retry a bad answer with the error as feedback, a few times and no more.
Key words
| Word | What it means |
|---|---|
| Repair loop | asking again after a bad answer, with feedback |
| Feedback | the faults, sent back in a new message |
| Retry limit | the most extra goes you allow |
| Fallback | what to do when the limit is reached |
| Message list | the conversation sent with each call |
| Give up | to stop and use the fallback |
The cells students fill in are left blank in this preview.
Send the faults back
To repair a reply, ask again with the whole conversation so far. Add the bad reply as a message from the assistant. Then add a message from the user that lists the faults and repeats the rule: reply with only the JSON.
The list of messages grows by two each time. The model sees what it said and what was wrong with it.
Worked example: three goes
| Go | Reply | Fault found | Sent next |
|---|---|---|---|
| 1 | Probably yes, tomatoes are fruit. | Not a JSON object | The fault, and the rule to reply with only the JSON |
| 2 | {"answer": "maybe"} | answer must be yes or no | The fault, and the rule again |
| 3 | {"answer": "yes"} | None | Nothing: accept it |
The cells students fill in are left blank in this preview.
A limit and a fallback
Each retry costs time and credit. So set a retry limit: the first go and at most two more. When the limit is reached, give up and use the fallback.
A fallback is a safe value such as "unknown", or an error a person will see. Write down that it happened, so you can see how often.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the repair loop, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' 'const allowed = [\"yes\", \"no\"];' 'const MAX_TRIES = 3; // the first go and at most two retries' '' 'function pullJson(text) {' ' const start = text.indexOf(\"{\");' ' const end = text.lastIndexOf(\"}\");' ' if (start === -1 || end < start) return null;' ' try { return JSON.parse(text.slice(start, end + 1)); } catch (error) { return null; }' '}' > repair3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the repair loop, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function problemsOf(text) {' ' const reply = pullJson(text);' ' if (reply === null) return [\"the reply is not a JSON object\"];' ' if (Object.prototype.hasOwnProperty.call(reply, \"answer\") === false) return [\"the key answer is missing\"];' ' if (allowed.includes(reply.answer) === false) return [\"answer must be yes or no\"];' ' return [];' '}' >> repair3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the repair loop, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function repairLoop(question, samples) {' ' const messages = [{ role: \"user\", content: question }];' ' for (let go = 1; go <= MAX_TRIES; go = go + 1) {' ' const reply = await ask(messages, samples, 60);' ' const found = problemsOf(reply);' ' console.log(\"Go \" + go + \": \" + (found.length === 0 ? \"ok\" : found.join(\"; \")));' ' if (found.length === 0) return { ok: true, reply: pullJson(reply), tries: go };' ' messages.push({ role: \"assistant\", content: reply });' ' messages.push({ role: \"user\", content: \"Fix these problems and reply with only the JSON: \" + found.join(\"; \") });' ' }' ' return { ok: false, reply: null, tries: MAX_TRIES };' '}' >> repair3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the repair loop, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const question = \"Is a tomato a fruit? Reply with a JSON object with the key answer, which is yes or no. Reply with only the JSON.\";' ' // Recorded samples, used only when no model answers: two bad replies, then a good one.' ' const samples = [\"Probably yes, tomatoes are fruit.\", JSON.stringify({ answer: \"maybe\" }), JSON.stringify({ answer: \"yes\" })];' ' const result = await repairLoop(question, samples);' ' const fallback = { answer: \"unknown\" };' ' console.log(\"Tries used:\", result.tries);' ' console.log(\"Result:\", JSON.stringify(result.ok ? { answer: result.reply.answer } : fallback));' '}' 'main();' >> repair3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the loop
docker exec -w /home/node/project agent node repair3.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| Go 1 | ok, if the model replied with clean JSON the first time |
| Go 2 and Go 3 | Only if go 1 failed: the fault, or ok |
| Tries used | A number from 1 to 3 |
| Result | The reply, or the fallback {"answer":"unknown"} |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think retrying until it works is safe. If the model never gets it right, a loop with no limit never ends, and every go costs time and credit. Always set a limit and decide the fallback first.