A search engine for a one page rulebook
Retrieval-augmented generation is a tool with a cost, and it is sometimes the wrong one. It adds chunking, vectors and a search that can miss the answer.
A bike club has one page of rules. A student builds a whole retrieval system for it. The page would have fitted in the prompt, and the search is now the part that can fail.
By the end of this lesson
Note
I can say when retrieval is the wrong tool.
Key words
| Word | What it means | Example |
|---|---|---|
| Context window | the most text a model reads in one go | |
| Token | a small piece of text a model counts | about 3/4 of a word |
| Budget | the most tokens you choose to spend | |
| Live data | facts that change every few minutes | |
| Retrieval | finding the chunks that fit a question |
The cells students fill in are left blank in this preview.
Retrieval is a trade, not a default
Retrieval lets a model use more text than fits in one prompt. That is its gain.
It costs work and adds risk. You have to chunk, build vectors and tune a search. The search can return the wrong chunk, or none, and then the answer is built from the wrong text. If you can skip it, skip it.
A small document fits whole in the prompt
A model reads a limited amount of text at once, its context window. Text is counted in tokens. For English, a token is about three quarters of a word. That is a rule of thumb, not an exact count.
Pick a budget of tokens for the context. If the whole document is under it, put it all in. There is nothing to search, so nothing to miss.
Worked example: six cases and their methods
| Case | Method | Why |
|---|---|---|
| A one page rulebook | Whole prompt | It fits, so there is nothing to search |
| A 40,000 word handbook | Retrieval | Too big, so find the few chunks that matter |
| The total fines for 17 late books | Calculate in code | A model can slip on arithmetic |
| The delay of the next bus | Live lookup | It changes every minute, so a store is stale |
| A policy that is 3 pages and rarely asked about | Whole prompt | It fits and the extra cost is small |
| A college intranet of thousands of pages | Retrieval | No prompt can hold it |
The cells students fill in are left blank in this preview.
Measure a document against a budget
The next script holds the bike club rules and counts the words. It turns words into tokens with the rule of thumb and compares them with a budget of 3000.
It does the same for a 40,000 word handbook, using the number only, and then builds the whole prompt for the rules. It calls no model.
Write the budget script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const rules = \"Rides start at 9 am from the old mill. Every rider wears a helmet. A group has at most 12 riders and one leader. The leader sets the pace and waits at every turn. Lights are needed on any ride that ends after sunset. Members pay 20 pounds a year. Guests may join two rides before they pay. Rides are cancelled in ice or heavy rain, and the leader tells everyone by 7 am. Nobody is left behind. Report any fall to the leader on the same day.\";' '// A rule of thumb for English: a token is about three quarters of a word.' 'function tokensFor(words) { return Math.round(words * 4 / 3); }' 'const BUDGET = 3000;' > whole.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the budget script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const documents = [' ' { name: \"Bike club rules\", words: rules.split(\" \").length },' ' { name: \"Library handbook\", words: 40000 },' '];' 'documents.forEach(function (one) {' ' const need = tokensFor(one.words);' ' const verdict = need <= BUDGET ? \"fits whole\" : \"needs retrieval\";' ' console.log(one.name + \": \" + one.words + \" words, about \" + need + \" tokens, \" + verdict);' '});' 'const prompt = \"Answer only from the club rules.\n\n\" + rules + \"\n\nQuestion: How many riders may be in a group?\";' 'console.log(\"The whole prompt is\", prompt.length, \"characters. No search is needed.\");' >> whole.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it and read the verdicts
docker exec -w /home/node/project agent node whole.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What it prints
| Line | What it shows |
|---|---|
| Bike club rules: 88 words, about 117 tokens, fits whole | Far under the budget |
| Library handbook: 40000 words, about 53333 tokens, needs retrieval | Far over it |
| The whole prompt is 520 characters | Small enough to send as it is |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think retrieval is always better than putting the document in the prompt. If the document fits, retrieval adds a search that can miss the answer. Use it when you have to, not because it sounds advanced.