A customer who changes their mind
A simulated user is a script that plays the customer in a conversation with your assistant. It matters because a dialogue has no single right reply, and typing test chats by hand does not scale.
Mina asks for a refund, names one order, then changes her mind. Does the assistant keep up?
By the end of this lesson
Note
I can test a dialogue with a scripted user and score the conversation.
Key words
| Word | What it means |
|---|---|
| Simulated user | a script that plays the customer |
| Persona | who the user is and how they write |
| Goal | what the user wants by the end |
| End condition | the rule that stops the dialogue |
| Transcript | the whole conversation, line by line |
The cells students fill in are left blank in this preview.
Write the user as a script
Give the user a persona, a goal and a list of lines. Add an end condition, such as a goodbye or a limit of turns, so the dialogue always stops.
A script gives the same user every run. A model that plays the user would vary, cost credit on every line and make a failure hard to repeat. Start with a script.
Worked example: Mina's four lines
| Line | What Mina says | What it tests |
|---|---|---|
| 1 | hi, refund please | does the assistant ask which order |
| 2 | order A17 | does it start the refund |
| 3 | actually make it order B02 | does it handle the change of mind |
| 4 | thanks, bye | does it end the dialogue |
The cells students fill in are left blank in this preview.
Score the transcript, not each reply
Check three things on the whole transcript. Was the goal reached? Were the turns within the limit? Did the action happen the right number of times?
The last check catches a quiet fault. An assistant that starts a refund for A17 and again for B02 reached the goal and refunded twice.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the dialogue driver, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '// A scripted user: a persona, a goal, the lines, and a change of mind at line 3.' 'const user = {' ' persona: \"Mina, in a hurry, writes short messages\",' ' goal: \"a refund for order B02\",' ' lines: [\"hi, refund please\", \"order A17\", \"actually make it order B02\", \"thanks, bye\"],' '};' '// A stand-in assistant made of rules. A real one would be the agent under test.' 'function assistant(text) {' ' const order = text.match(/[AB][0-9][0-9]/);' ' if (order) return \"Refund started for \" + order[0] + \".\";' ' if (text.includes(\"bye\")) return \"Goodbye.\";' ' return \"Which order number is it?\";' '}' > sim.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the dialogue driver, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function drive(maxTurns) {' ' const transcript = [];' ' for (let i = 0; i < user.lines.length && i < maxTurns; i = i + 1) {' ' transcript.push(\"user: \" + user.lines[i]);' ' const reply = assistant(user.lines[i]);' ' transcript.push(\"assistant: \" + reply);' ' if (reply.includes(\"Goodbye\")) break;' ' }' ' return transcript;' '}' 'const transcript = drive(6);' 'transcript.forEach(function (line) { console.log(line); });' 'const refunds = transcript.filter(function (line) { return line.includes(\"Refund started\"); });' >> sim.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the dialogue driver, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'console.log(\"Goal reached: \" + transcript.join(\" \").includes(\"Refund started for B02\"));' 'console.log(\"Turns used: \" + transcript.filter(function (line) { return line.startsWith(\"user\"); }).length);' 'console.log(\"Refunds started: \" + refunds.length + \" (the user wanted one)\");' 'async function main() {' ' const system = \"You are a support assistant for a shop. Ask for the order number before you refund anything. Keep replies short.\";' ' const sample = \"Sure. Which order number is it?\";' ' const text = await ask([{ role: \"system\", content: system }, { role: \"user\", content: user.lines[0] }], sample, 60);' ' console.log(\"Live reply: \" + text.trim());' ' console.log(\"Asks about the order: \" + text.toLowerCase().includes(\"order\"));' '}' 'main();' >> sim.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Drive the assistant with Mina
docker exec -w /home/node/project agent node sim.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the driver prints
| Line | What it shows |
|---|---|
| Transcript | Four user lines and four assistant lines |
| Goal reached | true |
| Turns used | 4 |
| Refunds started | 2, the user wanted one |
| Live reply | One possible output: a question about the order |
The cells students fill in are left blank in this preview.
Try it
Tip
In sim.js remove the third line, "actually make it order B02", and run again. Refunds started falls to 1, but the goal is no longer reached. Say why.
A common mistake
Warning
Many people think a dialogue passes when its last reply sounds right. A polite goodbye can follow two refunds. Score what the assistant did across the whole transcript: the goal, the actions and the turns.