Show where an answer came from
A citation is a marker in an answer that points at the source it came from. It matters because people trust an answer more when they can check it, and your program can check the markers.
A library chatbot says a late book costs ten pence a day [2]. The reader looks at source 2 and sees the same rule.
By the end of this lesson
Note
I can show which source an answer used.
Key words
| Word | What it means | Example |
|---|---|---|
| Source | a passage of text the answer may use | |
| Citation | a pointer from an answer to its source | |
| Marker | the number in brackets that makes a citation | [2] |
| Grounded | based on the sources you gave | |
| Unused source | a source that no marker names |
The cells students fill in are left blank in this preview.
Number the sources and ask for markers
Write each source into the prompt with a number: [1], [2], [3]. Then ask the model to use only those sources, and to write the number after each fact.
The sources here are passages you wrote into the script. Where real sources come from is a later module. The model's answer is grounded when it keeps to what you gave it.
Worked example: sources and an answer
| Part | Text |
|---|---|
| Source [1] | Members may borrow up to six books for three weeks. |
| Source [2] | A late book costs ten pence a day. |
| Question | How many books can I borrow, and what is the late fee? |
| Answer | You can borrow six books [1]. A late book costs ten pence a day [2]. |
The cells students fill in are left blank in this preview.
Check that every marker has a source
A model can write [4] when there are only three sources. Your program can catch that. Find every marker, then test each number against the list of source ids.
Also look for unused sources, and for sentences with no marker. They show what the answer left out.
A common mistake
Warning
Many people think a marker proves the answer is true. It only proves the source exists. The model can cite the wrong source for a fact. The check is a first filter, and a person still reads the source for anything that matters.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the citation check, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const SOURCES = [' ' { id: 1, text: \"Members may borrow up to six books for three weeks.\" },' ' { id: 2, text: \"A late book costs ten pence a day.\" },' ' { id: 3, text: \"The library opens at nine and closes at five on weekdays.\" },' '];' > cite5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the citation check, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function sourceText() {' ' return SOURCES.map(function (s) { return \"[\" + s.id + \"] \" + s.text; }).join(\"\n\");' '}' 'function markersIn(reply) {' ' const found = [];' ' const re = /\[(\d+)\]/g;' ' let match = re.exec(reply);' ' while (match) { found.push(Number(match[1])); match = re.exec(reply); }' ' return found;' '}' >> cite5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the citation check, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function check(reply) {' ' const ids = SOURCES.map(function (s) { return s.id; });' ' const markers = markersIn(reply);' ' const missing = markers.filter(function (n) { return ids.includes(n) === false; });' ' return { markers: markers, missing: missing, ok: markers.length > 0 && missing.length === 0 };' '}' 'function show(label, result) {' ' console.log(label + \" markers \" + JSON.stringify(result.markers) + \", missing \" + JSON.stringify(result.missing) + \", ok: \" + result.ok);' '}' >> cite5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the citation check, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const rule = \"Answer using only these numbered sources. After each fact write its source number in square brackets, like [1]. If the sources do not say, reply: I do not know.\n\" + sourceText();' ' const question = \"How many books can I borrow, and what is the late fee?\";' ' // Recorded sample, used only when no model answers.' ' const sample = \"You can borrow six books [1]. A late book costs ten pence a day [2].\";' ' const reply = await ask([{ role: \"system\", content: rule }, { role: \"user\", content: question }], sample, 80);' ' console.log(\"Reply:\", reply);' ' show(\"Check:\", check(reply));' ' show(\"Made-up reply:\", check(\"Loans last three weeks [4].\"));' '}' 'main();' >> cite5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask and check the markers
docker exec -w /home/node/project agent node cite5.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| Reply | Two short sentences, each ending with a marker such as [1] |
| Check: | markers [1,2], missing [], ok: true, if the reply kept to the sources |
| Made-up reply: | markers [4], missing [4], ok: false, every time |
The cells students fill in are left blank in this preview.