Prompt, retrieval or fine-tuning: how to choose
Three ways to change what a model-backed feature does are a better prompt, retrieval (RAG, retrieval-augmented generation) and fine-tuning. They fix different problems, and they cost very different amounts. It matters because teams often pick the most expensive one first.
A support team wants answers in their house tone, from this week's price list, in a fixed format. That is three needs, and they may not share one fix.
By the end of this lesson
Note
I can choose between a better prompt, retrieval and fine-tuning for a need.
Key words
| Word | What it means |
|---|---|
| Prompt | the instructions and examples sent with a call |
| Retrieval | fetching documents and sending them with the call |
| Fine-tuning | training a model further on your own examples |
| Training example | an input paired with the output you want |
| Knowledge | facts the answer depends on |
The cells students fill in are left blank in this preview.
Fine-tuning is not run here
Warning
The lab cannot fine-tune a model. This lesson is a decision exercise. It gives the questions to ask and a teaching rule. It does not claim what a fine-tune would achieve for you: measure that on your own cases.
Ask what kind of problem it is
Ask four questions in order. Does the answer depend on facts that are private or change? Does the model fail on form, such as tone or format? Have you tried a clear prompt with examples? Do you have enough examples and a way to measure?
Facts point to retrieval. Form points to a prompt first, and fine-tuning only when a prompt fails and the examples exist.
Worked example: five needs
| Need | First question | Likely first step |
|---|---|---|
| A friendly tone | Does a prompt with examples fail? | Prompt, then fine-tuning if it fails and examples exist |
| Prices that change weekly | Do the facts change? | Retrieval: fine-tuned facts go stale |
| A strict output format | Can examples show it? | Prompt, with a check that parses the output |
| Answers from private manuals | Are the facts private? | Retrieval, with sources shown |
| A lower cost per call | What is the prompt's size? | Measure first: a shorter prompt or a smaller model |
The cells students fill in are left blank in this preview.
What each choice costs and cannot fix
A prompt is cheap to change and easy to undo, but it uses tokens on every call. Retrieval adds a store and a search, and fails when the search misses. Fine-tuning needs examples, a training run and a new model to look after, and it does not keep facts fresh.
None of them fixes a missing requirement or a bad evaluation. Write the cases that decide success first.
A common mistake
Warning
Many people think fine-tuning teaches a model new facts. It mostly shapes style and format. Facts that change belong in retrieval, where you can update a document and see the answer change the same day.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the decision rule, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const cases = [' ' { need: \"a friendly tone\", privateDocs: false, factChanges: false, promptFails: true, examples: 800 },' ' { need: \"prices that change weekly\", privateDocs: false, factChanges: true, promptFails: false, examples: 0 },' ' { need: \"a strict output format\", privateDocs: false, factChanges: false, promptFails: false, examples: 40 },' ' { need: \"answers from our manuals\", privateDocs: true, factChanges: false, promptFails: true, examples: 60 },' ' { need: \"a lower cost per call\", privateDocs: false, factChanges: false, promptFails: false, examples: 0 },' '];' > decide.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the decision rule, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function choose(c) {' ' const retrieve = c.privateDocs || c.factChanges;' ' const tune = c.promptFails && c.examples >= 500;' ' if (retrieve && tune) return \"retrieval and fine-tune\";' ' if (retrieve) return \"retrieval\";' ' if (tune) return \"fine-tune\";' ' if (c.promptFails) return \"collect examples\";' ' return \"prompt\";' '}' 'cases.forEach(function (c) { console.log(c.need + \":\", choose(c)); });' >> decide.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the decision rule, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const text = \"Reply with only JSON with keys product and price for: Bike, 120 pounds.\";' ' const sample = JSON.stringify({ product: \"Bike\", price: 120 });' ' const answer = await ask([{ role: \"user\", content: text }], sample, 40);' ' let valid = true;' ' try { JSON.parse(answer); } catch (error) { valid = false; }' ' console.log(\"Prompt first:\", answer);' ' console.log(\"Valid JSON:\", valid);' '}' 'main();' >> decide.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Apply the rule to five cases
docker exec -w /home/node/project agent node decide.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The rule is a teaching rule
The Run applies a rule that is easy to read, not a measurement. The threshold of 500 examples is an invented number for the exercise. Your own threshold comes from your cases and your evaluation.
The live call tries the cheapest step first: a prompt that asks for JSON, then a parse to check it.
One possible output
| Line | What it shows |
|---|---|
| a friendly tone: fine-tune | The prompt failed and 800 examples exist |
| prices that change weekly: retrieval | The facts change |
| a lower cost per call: prompt | Nothing failed, so measure the prompt first |
| Valid JSON | true or false: the answer was parsed, not read |
The cells students fill in are left blank in this preview.