One search was not enough
Agentic retrieval means an agent decides to search again when the first results do not answer the question. It matters because a real question often needs two looks: the first finds the topic and the second the detail.
A user asks how long they are locked out. The first search finds password rules, not the lock-out time. The agent searches again.
By the end of this lesson
Note
I can let an agent search more than once, with a stop rule and a limit, and score its answers on known questions.
Key words
| Word | What it means |
|---|---|
| Agentic retrieval | search the agent may repeat and change |
| Stop rule | the condition that ends the search loop |
| Search limit | the most searches one question may use |
| Recall | the share of the needed sources that were found |
| Supported answer | an answer whose sources were really retrieved |
| Seen | the sources found so far |
The cells students fill in are left blank in this preview.
A search is a tool the loop reuses
An agent loop calls a tool, reads the result and decides the next step, within a budget of steps. Here the tool is search. The agent searches, reads what came back, and either answers or searches again with a new query.
Two things need care. The loop must end. And the final answer must rest on what was really found.
Three ways to stop
Stop when the agent answers. Stop when a search adds nothing new, because searching again with the same words will not help. Stop at a limit on searches, set by cost and time.
Record why the loop stopped: answered, stopped, limit or no answer. The reason tells you what to fix.
Worked example: three runs of the same question
| Run | Searches | Ends with | Supported |
|---|---|---|---|
| Good | 2 | answered | yes |
| Stuck | 2 | stopped, nothing new | no |
| Limit of 1 | 1 | stopped, limit | no |
The cells students fill in are left blank in this preview.
Check the answer rests on what was found
An answer lists its sources. It is supported only if every source is among the pages the searches returned. A source the agent never saw was invented.
Measure recall too: of the pages a question needs, how many did the searches find? Score both over questions with known sources.
A common mistake
Warning
Many people think an agent that cites a source has used it. A citation is only text. Check that each cited id was returned by a search in this run, or an invented source will pass as a real one.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the search loop, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const index = {' ' \"password rules\": [\"p2\", \"p1\"],' ' \"lock out time\": [\"p3\"],' ' \"reset password\": [\"p1\"],' '};' 'const docs = { p1: \"Reset a password from the sign in page.\", p2: \"A password needs ten letters.\", p3: \"After five wrong passwords an account is locked for fifteen minutes.\" };' > agentic1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the search loop, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function run(steps, max) {' ' const seen = [];' ' let n = 0;' ' for (const s of steps) {' ' if (\"answer\" in s) {' ' const ok = s.sources.length > 0 && s.sources.every(function (p) { return seen.includes(p); });' ' return \"answered, supported: \" + (ok ? \"yes\" : \"no\");' ' }' ' if (n === max) return \"stopped: the limit of \" + max + \" searches\";' ' n++;' ' const found = index[s.search] || [];' ' const fresh = found.filter(function (p) { return seen.includes(p) === false; });' ' fresh.forEach(function (p) { seen.push(p); });' ' console.log(\" search \" + n + \": \" + s.search + \" -> \" + (found.join(\" \") || \"nothing\") + \", new \" + fresh.length);' ' if (fresh.length === 0) return \"stopped: nothing new\";' ' }' ' return \"no answer\";' '}' >> agentic1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the search loop, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const good = [{ search: \"password rules\" }, { search: \"lock out time\" }, { answer: \"Fifteen minutes.\", sources: [\"p3\"] }];' 'const stuck = [{ search: \"reset password\" }, { search: \"reset password\" }, { answer: \"Unsure.\", sources: [\"p1\"] }];' 'console.log(\"good run:\"); console.log(run(good, 3));' 'console.log(\"stuck run:\"); console.log(run(stuck, 3));' 'console.log(\"limit run:\"); console.log(run(good, 1));' 'async function main() {' ' const text = \"Question: how long am I locked out. First search found: \" + docs.p2 + \" Write one better search query of 5 words or fewer. Reply with the query only.\";' ' const reply = await ask([{ role: \"user\", content: text }], \"account lock out time\", 20);' ' console.log(\"a model suggests:\", reply.text.trim());' '}' 'main();' >> agentic1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the three loops
docker exec -w /home/node/project agent node agentic1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
This is a stand-in
The loop follows recorded steps, so every student sees the same three runs. A real agent chooses its own steps, and the model call here only suggests a query, which the Run prints and does not follow.
The index is three entries. A real one would be a hybrid search over many documents, with an access filter.
What the Run prints
| Run | Last line |
|---|---|
| good run | answered, supported: yes |
| stuck run | stopped: nothing new |
| limit run | stopped: the limit of 1 searches |
| model suggestion | One possible output: account lock out time |
The cells students fill in are left blank in this preview.