Try a prompt before you train
Adapting a model means changing the model itself with your own examples. It costs time, data and care. Many problems that look like they need it are fixed by a better prompt.
A team that trains first, and tests a prompt second, often pays for a fix it did not need.
By the end of this lesson
Note
I can show that a prompt cannot fix a problem before I reach for training.
Key words
| Word | What it means |
|---|---|
| Baseline | the results you get before any change |
| Case | one input with a known right answer |
| Pass rate | the percent of cases answered right |
| Adapting | changing a model with your own examples |
| Target | the pass rate you need to be happy |
The cells students fill in are left blank in this preview.
Start with a baseline
Write down a set of cases first. A case is one input with the answer you want. Run your current prompt over every case and count the passes.
That count is the baseline. Without it you cannot say a change helped. Keep the cases fixed, so every later run is fair.
Worked example: a ticket sorter
| Kind of ticket | Cases | Baseline passes | Better prompt passes |
|---|---|---|---|
| billing | 2 | 2 | 2 |
| delivery | 2 | 1 | 2 |
| refund | 2 | 0 | 2 |
| tone | 2 | 0 | 0 |
| all | 8 | 3 (37%) | 6 (75%) |
The cells students fill in are left blank in this preview.
What a prompt can and cannot fix
A prompt can fix a missing instruction, a missing example or a wrong output form. It is cheap and you can undo it in a second.
A prompt struggles when the model must follow a style it was never shown, or know facts the window cannot hold. Those are the cases where adapting may be worth it. Module 79 teaches the prompt craft itself.
A common mistake
Warning
Many people think a wrong answer means the model needs training. Often the prompt never said what was wanted. Check the prompt and a few examples first, and train only if a fair test shows no real gain.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the baseline report, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '// Recorded results: 1 means the sorter put the ticket in the right pile.' 'const kinds = [\"billing\", \"billing\", \"delivery\", \"delivery\", \"refund\", \"refund\", \"tone\", \"tone\"];' 'const baseline = [1, 1, 1, 0, 0, 0, 0, 0];' 'const prompted = [1, 1, 1, 1, 1, 1, 0, 0];' > adapt1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the baseline report, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function rate(list) {' ' let ok = 0;' ' for (const r of list) ok = ok + r;' ' return Math.floor((ok * 100) / list.length);' '}' 'function failing(list) {' ' const out = [];' ' list.forEach(function (r, i) {' ' if (r === 0 && out.includes(kinds[i]) === false) out.push(kinds[i]);' ' });' ' return out;' '}' >> adapt1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the baseline report, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"recorded runs, 8 cases\");' ' console.log(\"baseline \" + rate(baseline) + \"%, still failing: \" + failing(baseline).join(\", \"));' ' console.log(\"better prompt \" + rate(prompted) + \"%, still failing: \" + failing(prompted).join(\", \"));' ' const q = \"In one sentence, can a prompt teach a model a house writing style it has never seen?\";' ' const sample = \"A prompt can show a few examples of the style, but it cannot change what the model does by default.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 40);' ' console.log(\"model: \" + reply.text);' '}' 'main();' >> adapt1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the baseline report
docker exec -w /home/node/project agent node adapt1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The results are recorded, not live
The two lists of results are recorded and written into the script. They are not made by a model, so every student sees the same numbers. Only the last line calls the lab's model, and with no model it prints a Recorded sample.