The right answer by a bad path
A trajectory is the list of steps an agent took to reach its answer. Scoring it matters because an agent can give the right answer after wasting calls, or by luck, and you pay for every step.
The shop has a support agent that refunds orders. Its answers look fine. Its paths do not.
By the end of this lesson
Note
I can score the steps an agent took against a reference path.
Key words
| Word | What it means |
|---|---|
| Trajectory | the steps an agent took, in order |
| Reference path | the steps a good run is expected to take |
| Wasted call | a step that repeats an earlier one |
| Step budget | the most steps a run may take |
| Path score | points for how the run got there |
The cells students fill in are left blank in this preview.
Score the path, not only the answer
A final-answer check asks one question: was the reply right? A path check asks how the agent got there. It looks at the trajectory and compares it with a reference path.
A path can show four faults. A step is missing. A wasted call repeats one. A step is out of order. Or there are too many steps for the step budget.
Worked example: four recorded runs
| Run | Steps taken | Answer | Path faults |
|---|---|---|---|
| A | lookup, refund, send | right | none |
| B | lookup, lookup, search, refund, send | right | a repeat, over budget |
| C | refund, lookup, send | right | refund before lookup |
| D | lookup, send | wrong | refund missing, so out of order too |
The cells students fill in are left blank in this preview.
Turn the faults into one score
Give each fault a fixed cost and subtract it from 100. Here a missing step costs 25, a wasted call 10, going over the budget 20 and a wrong order 15.
Choose the costs before you look at the runs. The reference is not the only good path, so use order only where order matters.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the recorded runs
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const reference = [\"lookup_order:A17\", \"refund:A17\", \"send_message:A17\"];' '// Recorded runs of a support agent. Each step is a tool and its argument.' 'const runs = [' ' { name: \"Run A\", answer: \"ok\", steps: [\"lookup_order:A17\", \"refund:A17\", \"send_message:A17\"] },' ' { name: \"Run B\", answer: \"ok\", steps: [\"lookup_order:A17\", \"lookup_order:A17\", \"search_help:refund\", \"refund:A17\", \"send_message:A17\"] },' ' { name: \"Run C\", answer: \"ok\", steps: [\"refund:A17\", \"lookup_order:A17\", \"send_message:A17\"] },' ' { name: \"Run D\", answer: \"wrong\", steps: [\"lookup_order:A17\", \"send_message:A17\"] },' '];' 'module.exports = { reference, runs };' > paths.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the path scorer, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { reference, runs } = require(\"./paths\");' 'const { ask } = require(\"./model\");' 'const budget = 4;' 'function wasted(steps) {' ' const seen = [];' ' let count = 0;' ' steps.forEach(function (step) {' ' if (seen.includes(step)) count = count + 1;' ' else seen.push(step);' ' });' ' return count;' '}' 'function missing(steps) {' ' return reference.filter(function (step) { return steps.includes(step) === false; });' '}' > score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the path scorer, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function inOrder(steps) {' ' let at = 0;' ' steps.forEach(function (step) {' ' if (at < reference.length && step === reference[at]) at = at + 1;' ' });' ' return at === reference.length;' '}' 'function score(steps) {' ' let points = 100 - 25 * missing(steps).length - 10 * wasted(steps);' ' if (steps.length > budget) points = points - 20;' ' if (inOrder(steps) === false) points = points - 15;' ' return Math.max(points, 0);' '}' >> score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the path scorer, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const weak = [];' 'runs.forEach(function (run) {' ' const points = score(run.steps);' ' console.log(run.name + \": \" + points + \" points, steps \" + run.steps.length + \" of \" + budget + \", missing \" + missing(run.steps).length + \", wasted \" + wasted(run.steps) + \", in order \" + inOrder(run.steps) + \", answer \" + run.answer);' ' if (run.answer === \"ok\" && points < 100) weak.push(run.name);' '});' 'console.log(\"Right answer, weak path: \" + weak.join(\", \"));' >> score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the path scorer, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const prompt = \"A customer asks for a refund on order A17. Tools: lookup_order, refund, send_message, search_help. Reply with only the name of the tool you would call first.\";' ' const text = await ask([{ role: \"user\", content: prompt }], \"lookup_order\", 20);' ' console.log(\"Live first step: \" + text.trim());' ' console.log(\"Matches the reference: \" + text.includes(\"lookup_order\"));' '}' 'main();' >> score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Score the four runs
docker exec -w /home/node/project agent node score.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the scorer prints
| Line | What it shows |
|---|---|
| Run A | 100 points, no faults |
| Run B | 70 points: one wasted call and 5 steps of 4 |
| Run C | 85 points: not in order |
| Run D | 60 points: refund missing (25) and out of order (15), answer wrong |
| Weak path | Run B and Run C |
| Live first step | One possible output: lookup_order |
The cells students fill in are left blank in this preview.
Try it
Tip
In score.js change the budget from 4 to 5, then score again. Run B should gain 20 points. Decide whether that makes it a good path.
A common mistake
Warning
Many people think the reference path is the only correct one and mark every other order as a fault. Agents often have two good paths. Score order only where one step needs the result of another, and allow a second reference when two orders both work.