Sure, or just loud?
A weather feature says "73.4% chance of rain". Another says "probably rain". Which one is more honest if the forecast is only good to the nearest ten?
The first looks exact but is not. Showing how sure an answer is means choosing words and numbers that match what you really know.
By the end of this lesson
Note
I can show how sure an answer is without false precision.
Key words
| Word | What it means | Example |
|---|---|---|
| Score | a number for how sure an answer is | 64 |
| Label | a word chosen from a score | medium |
| Cut point | the score where a label changes | |
| False precision | a number with more digits than you know | |
| Source | the text an answer came from |
The cells students fill in are left blank in this preview.
A word is easier to use than 64
A score such as 64 means little to most people. A word such as "medium" tells them what to do: check it.
Pick the cut points once, write them down and use them everywhere. Then "high" means the same on every screen.
Do not show digits you do not have
A score of 64 from a rough check is not really 64 and not 63. Show "about 60", or just the label.
Also show where an answer came from. A source lets a person check. When the score is very low, the honest design refuses and says why.
A common mistake
Warning
Many people think the model's own confidence is a real probability. It is not always. Treat a score as a rough guide. Check labels against known cases. Keep the cut points steady, so a higher score never gives a lower label.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the label maker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const CUTS = [[80, \"high\"], [50, \"medium\"]];' 'const ANSWERS = [' ' { id: \"q1\", score: 91 },' ' { id: \"q2\", score: 80 },' ' { id: \"q3\", score: 64 },' ' { id: \"q4\", score: 49 },' ' { id: \"q5\", score: 12 },' '];' > showing.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the label maker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function label(score) {' ' for (const c of CUTS) {' ' if (score >= c[0]) return c[1];' ' }' ' return \"low\";' '}' 'function blurred(score) {' ' return Math.floor((score + 5) / 10) * 10;' '}' >> showing.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the label maker, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const counts = { high: 0, medium: 0, low: 0 };' ' for (const a of ANSWERS) {' ' counts[label(a.score)]++;' ' console.log(a.id + \" score \" + a.score + \" shown about \" + blurred(a.score) + \" label \" + label(a.score));' ' }' ' console.log(\"counts \" + JSON.stringify(counts));' ' const msg = \"Write one short line telling a student you are not sure and they should check the source.\";' ' const reply = await ask([{ role: \"user\", content: msg }], \"I am not sure about this. Please check the source.\", 30);' ' console.log(\"refusal line: \" + reply.text);' '}' 'main();' >> showing.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the label maker
docker exec -w /home/node/project agent node showing.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| q1 score 91 shown about 90 label high | High, and the number is blurred |
| q2 score 80 shown about 80 label high | 80 is on the cut point, so high |
| q4 score 49 shown about 50 label low | The blur says 50 but the label says low |
| counts {"high":2,"medium":1,"low":2} | Two of each end and one in the middle |
The cells students fill in are left blank in this preview.