Which feature needs the most care
A risk rating says how much harm an AI feature could do and so how much care it needs before it ships. It matters because a menu planner and a CV screener may use the same model, yet they are not the same risk.
This lesson is general information and not legal advice. Check the rules that apply to you, and ask a lawyer when a decision about a person is involved.
By the end of this lesson
Note
I can rate how risky an AI feature is and say who is accountable for it.
Key words
| Word | What it means |
|---|---|
| Risk rating | a level that says how much care a feature needs |
| Automatic decision | a result that no person checks before it acts |
| Reversible | a wrong result can be put right |
| Owner | the named person who answers for the feature |
| Accountable | able to be asked why, and expected to fix it |
The cells students fill in are left blank in this preview.
Four questions give a rating
Ask four things about a feature. Does it act on people? Does a person check each result before it acts? Can a wrong result be put right? How many people does it reach?
Each risky answer adds points: it acts on people, nobody checks it, it cannot be undone, or it reaches over 1000 people. The total gives a level. A rating is a starting point for a conversation, not a verdict. It stops a risky feature being treated like a toy.
Worked example: two features
| Feature | People | Automatic | Not undoable | Over 1000 users | Score and level |
|---|---|---|---|---|---|
| Order tracker | no | yes: 2 | no | no | 2, low |
| CV screener | yes: 2 | yes: 2 | yes: 2 | no | 6, high |
The cells students fill in are left blank in this preview.
Every feature has a named owner
A role such as "the data team" is not an owner. An owner is one person who is asked when the feature does harm, and who may switch it off.
Some places have rules about decisions made about people by a system alone. They often ask that a person can review the decision. This is a summary, not legal advice: find the rules that apply to you.
A common mistake
Warning
Many people think the model decides the risk, so a safe model means a safe feature. The risk comes from what the feature does, to whom, and whether it can be undone. Rate the feature, never the model.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk rater, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' '// Invented features. Each says what it does, to whom, and whether it can be undone.' 'const features = [' ' { name: \"Order tracker\", people: false, automatic: true, undo: true, users: 500 },' ' { name: \"CV screener\", people: true, automatic: true, undo: false, users: 300 },' ' { name: \"Study helper\", people: true, automatic: false, undo: true, users: 2000 },' ' { name: \"Refund bot\", people: true, automatic: true, undo: true, users: 800 },' '];' 'const owners = { \"CV screener\": \"Priya Shah\", \"Refund bot\": \"\" };' > risk.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk rater, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function score(f) {' ' let s = 0;' ' if (f.people) s = s + 2;' ' if (f.automatic) s = s + 2;' ' if (f.undo === false) s = s + 2;' ' if (f.users > 1000) s = s + 1;' ' return s;' '}' 'function level(s) {' ' if (s >= 5) return \"high\";' ' if (s >= 3) return \"medium\";' ' return \"low\";' '}' >> risk.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk rater, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' features.forEach(function (f) {' ' const s = score(f);' ' const who = owners[f.name] || \"nobody\";' ' console.log(f.name + \": score \" + s + \", \" + level(s) + \", owner: \" + who);' ' });' ' const q = \"In two sentences: one harm a CV screening feature could do to a person, and who should answer for it.\";' ' const r = await ask([{ role: \"user\", content: q }], \"It could reject a good candidate by mistake and nobody would see it. A named person in the company must answer for it.\", 80);' ' console.log(\"One possible answer: \" + r.text);' '}' 'main();' >> risk.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the risk rater
docker exec -w /home/node/project agent node risk.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| Order tracker: score 2, low, owner: nobody | Low risk, so no owner is not urgent |
| CV screener: score 6, high, owner: Priya Shah | High risk and a named owner |
| Study helper: score 3, medium, owner: nobody | Another gap: medium risk with no owner |
| Refund bot: score 4, medium, owner: nobody | A gap: medium risk and nobody answers for it |
| One possible answer: ... | Your own call. It differs each run, or says Recorded sample |
The cells students fill in are left blank in this preview.