Sorting the support inbox
Classifying text with a model means asking it to pick one label from a fixed list for each message. It matters because a program can route a label, but it cannot route a free sentence.
A shop wants each support message sent to billing, to the app team or to a general desk. A model can read the message and name the desk. The question is how often it is right.
By the end of this lesson
Note
I can label text from a fixed list and measure how often the labels are right.
Key words
| Word | What it means | Example |
|---|---|---|
| Label | one word that names the kind of a text | billing |
| Allowed labels | the fixed list a label must come from | |
| Unknown | the label for a text that fits none | |
| Known answer | the label a person chose in advance | |
| Accuracy | the share of labels that match the known answer | |
| Test cases | texts with known answers, used to measure |
The cells students fill in are left blank in this preview.
Give a fixed list and an unknown label
Put the allowed labels in the prompt and say: reply with only the word. Add a way out. If no label fits, the model should say unknown, so it is not forced to pick a wrong one.
Then clean what comes back. A reply may have a capital letter, a full stop or extra words. Turn it into one allowed label, or into unknown.
Worked example: cleaning five replies
| Reply | Cleaned label | Why |
|---|---|---|
| billing | billing | Already an allowed label |
| Bug. | bug | Capital letter and full stop removed |
| "other" | other | Quotation marks removed |
| It is a bug | unknown | Several words, not one label |
| refund | unknown | Not on the allowed list |
The cells students fill in are left blank in this preview.
Measure with known answers
To know how good a labeller is, test it. Write test cases first: texts with a known answer that a person chose. Run the model on each, clean the label and compare.
Accuracy is the right labels divided by all labels, as a whole percentage. Then read the wrong ones. They show what to fix in the prompt.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the labeller, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' 'const allowed = [\"billing\", \"bug\", \"other\"];' '// Cases with known answers, written by a person before the model is asked.' 'const cases = [' ' { text: \"I was charged twice for my order.\", expected: \"billing\" },' ' { text: \"The app closes when I press Pay.\", expected: \"bug\" },' ' { text: \"What time do you open on Sunday?\", expected: \"other\" },' ' { text: \"My refund has not arrived.\", expected: \"billing\" },' '];' > label5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the labeller, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Recorded samples, used only when no model answers. Two are messy on purpose.' 'const samples = [\"billing\", \"Bug.\", \"other\", \"refund\"];' >> label5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the labeller, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Turn any reply into one allowed label, or unknown.' 'function cleanLabel(reply) {' ' let text = reply.trim().toLowerCase();' ' while (text.length > 0 && (text.endsWith(\".\") || text.endsWith(\" \"))) text = text.slice(0, -1);' ' return allowed.includes(text) ? text : \"unknown\";' '}' >> label5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the labeller, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' let right = 0;' ' let unknown = 0;' ' for (let at = 0; at < cases.length; at = at + 1) {' ' const prompt = \"Label this support message with exactly one word from this list: billing, bug, other. If none fits, reply unknown. Reply with only the word.\n\nMessage: \" + cases[at].text;' ' const reply = await ask([{ role: \"user\", content: prompt }], samples[at], 10);' ' const label = cleanLabel(reply);' ' if (label === cases[at].expected) right = right + 1;' ' if (label === \"unknown\") unknown = unknown + 1;' ' console.log((at + 1) + \" \" + label + \" (expected \" + cases[at].expected + \") \" + (label === cases[at].expected ? \"right\" : \"wrong\"));' ' }' ' console.log(\"Accuracy: \" + right + \" of \" + cases.length + \" (\" + Math.floor(right * 100 / cases.length) + \" percent)\");' ' console.log(\"Unknown labels: \" + unknown);' '}' 'main();' >> label5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it on four messages
docker exec -w /home/node/project agent node label5.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| 1 to 4 | A label, the expected label, and right or wrong |
| Accuracy | For example 3 of 4 (75 percent), or 4 of 4 |
| Unknown labels | How many replies could not be cleaned to an allowed label |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think 100 percent on a few cases means the labeller is good. A handful of cases proves little. Use many cases, include awkward ones and read the wrong answers.