An answer with its sources
Retrieval-augmented generation (RAG) means finding passages first and then asking a language model to answer from them alone. It matters because a model that answers from memory can sound sure and be wrong, while one that shows its sources can be checked.
A library visitor asks when it opens on Saturday. The site finds two passages, asks a model and shows the answer with the passage it came from.
By the end of this lesson
Note
I can build a prompt from what was found and answer only from it.
Key words
| Word | What it means | Example |
|---|---|---|
| Context | the passages you put in the prompt | |
| Prompt | the text you send to a model | |
| Citation | a marker that points to a passage | [1] |
| Grounded | based only on the passages given | |
| Refusal | an answer that says it does not know |
The cells students fill in are left blank in this preview.
Put the passages in the prompt
The model has not read your documents. So the prompt carries them. Number each passage so the model can point to one. Then add the question and the rules.
Keep the passages few and short. Every word you add is paid for, and a long prompt makes the answer harder to check.
Worked example: the parts of the prompt
| Part | Example | What it is for |
|---|---|---|
| Rule | Answer only from the passages below | Keeps the answer grounded |
| Citation rule | Cite each passage you use like [1] | Makes the answer checkable |
| Way out | Otherwise reply exactly: I do not know | Stops a made-up answer |
| Passages | [1] On Saturday the library opens at 10 am | The only facts allowed |
| Question | When does the library open on Saturday? | What to answer |
The cells students fill in are left blank in this preview.
Check what comes back
A prompt asks. It does not force. The model may cite a passage that does not exist, add a fact that was not given or forget to cite at all.
So the program checks. It finds the citation markers and looks each one up. An answer that cites nothing is not shown as sourced.
Worked example: what to do with each reply
| The reply | What the program does |
|---|---|
| Cites [1] and passage 1 exists | Show it with the source of passage 1 |
| Cites [7] and there are two passages | Do not trust it |
| Cites nothing | Do not show it as sourced |
| I do not know. | Show that, with no sources |
The cells students fill in are left blank in this preview.
Ask once and check the reply
The next Runs write two files, then make one short call. The first file, model.js, sends a prompt to the lab's model. If the model does not answer, it prints Recorded sample and uses a reply written in the script.
The second file, answer.js, builds the prompt, asks, finds the citations and says whether the reply refused. Nothing personal goes in a prompt.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the answer script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '// The passages a retriever found, written here so the Run costs one short call.' 'const passages = [' ' { source: \"hours-weekend\", text: \"On Saturday the library opens at 10 am and closes at 2 pm. It is closed on Sunday.\" },' ' { source: \"hours-week\", text: \"The library opens at 9 am and closes at 6 pm from Monday to Friday.\" },' '];' 'const question = \"When does the library open on Saturday?\";' 'function buildPrompt(question, passages) {' ' const lines = [\"Answer only from the passages below. Cite each passage you use like [1]. If the passages do not hold the answer, reply exactly: I do not know.\", \"\"];' ' passages.forEach(function (one, i) { lines.push(\"[\" + (i + 1) + \"] \" + one.text); });' ' lines.push(\"\", \"Question: \" + question);' ' return lines.join(\"\n\");' '}' > answer.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the answer script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Recorded sample, used only when no model answers.' 'const sample = \"It opens at 10 am on Saturday [1].\";' 'function citedSources(reply) {' ' const found = [];' ' (reply.match(/\[(\d+)\]/g) || []).forEach(function (mark) {' ' const one = passages[Number(mark.slice(1, -1)) - 1];' ' if (one && found.indexOf(one.source) < 0) found.push(one.source);' ' });' ' return found;' '}' 'async function main() {' ' const prompt = buildPrompt(question, passages);' ' console.log(prompt);' ' const reply = await ask([{ role: \"user\", content: prompt }], sample, 120);' ' console.log(\"Answer:\", reply);' ' const cited = citedSources(reply);' ' console.log(\"Sources:\", cited.length > 0 ? cited.join(\", \") : \"none cited\");' ' console.log(\"Says I do not know:\", /i do not know/i.test(reply) ? \"yes\" : \"no\");' '}' 'main();' >> answer.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it and read the answer
docker exec -w /home/node/project agent node answer.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| The prompt | The rules, two numbered passages and the question |
| Answer | It opens at 10 am on Saturday [1]. |
| Sources | hours-weekend |
| Says I do not know | no |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think that telling a model to use only the passages makes it do so. The rule lowers the chance of a made-up answer. It does not remove it. Keep the check, and change the question to one the passages cannot answer to see the way out work.