Where could this assistant go wrong
The main risks of a large language model (LLM) application are injection, unsafe output, too much power, leaked data and overuse. Each one lives in a particular place in your design, so you can look for it there.
A team ships a support bot and tests the happy path. A review asks a different question: for each box in the diagram, what comes in, and what can it do?
By the end of this lesson
Note
I can name the main risks of a model-backed application and say where each one lives in my design.
Key words
| Word | What it means |
|---|---|
| Unsafe output handling | using a reply as code or markup without checking it |
| Excessive agency | a model that can do more than its task needs |
| Data leak | private text reaching a place it should not |
| Overuse | requests or cost growing past what you planned |
| Trust boundary | a line where text from outside enters your program |
The cells students fill in are left blank in this preview.
Every risk lives in a place
Draw your design as boxes. At each box, ask what text comes in and what the box can do with it. A risk is a mismatch: untrusted text going somewhere with power.
The Open Worldwide Application Security Project (OWASP) publishes a list of top risks for these applications. Read it for the full set. This lesson uses five of them, in our own words.
Worked example: five risks and their places
| Risk | Where it lives | A made-up example | A first check |
|---|---|---|---|
| Injection | Where text enters the prompt | A page says to ignore the rules | Separate the data and check the reply |
| Unsafe output | Where the reply is used | A reply with a script tag put in a page | Escape it, or parse it strictly |
| Excessive agency | Where a tool runs | A helper bot that can delete folders | An allow-list and approval |
| Data leak | The prompt, the logs and the reply | A key pasted into a prompt and logged | Keep secrets and personal data out |
| Overuse | The route that calls the model | One user sends a flood of requests | A limit per user and per day |
The cells students fill in are left blank in this preview.
A reply is untrusted input
A model writes text that may repeat what an attacker put in the data. So treat a reply like a form field typed by a stranger.
If you show it in a page, escape it. Never paste it straight into a query or a command. Parse it into a fixed shape and check every field first.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk map script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' '// A small design written as data: each part and what flows into it.' 'const design = [' ' { part: \"chat box\", input: \"typed text\" },' ' { part: \"document loader\", input: \"pages from other people\" },' ' { part: \"prompt builder\", input: \"a secret key and the text\" },' ' { part: \"reply page\", input: \"model text shown as HTML\" },' ' { part: \"tool runner\", input: \"model text that picks an action\" },' ' { part: \"api route\", input: \"any number of requests\" },' '];' > risks.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk map script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Our own table: the kind of input, the risk it brings and the check.' 'const rules = {' ' \"typed text\": [\"injection\", \"separate it and check the reply\"],' ' \"pages from other people\": [\"indirect injection\", \"mark it as data and check the reply\"],' ' \"a secret key and the text\": [\"data leak\", \"keep secrets out of prompts\"],' ' \"model text shown as HTML\": [\"unsafe output\", \"escape it before showing it\"],' ' \"model text that picks an action\": [\"excessive agency\", \"allow-list the tools\"],' ' \"any number of requests\": [\"overuse\", \"limit per user and per day\"],' '};' 'for (const item of design) {' ' const rule = rules[item.input];' ' console.log(item.part + \" -> \" + rule[0] + \". Check: \" + rule[1] + \".\");' '}' >> risks.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the risk map script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function escapeHtml(text) {' ' const quote = String.fromCharCode(34);' ' return text.split(\"&\").join(\"&\").split(\"<\").join(\"<\")' ' .split(\">\").join(\">\").split(quote).join(\""\");' '}' 'async function main() {' ' const ask1 = [{ role: \"user\", content: \"Write one short sentence welcoming a visitor to a bakery website.\" }];' ' const reply = await ask(ask1, \"Welcome to our bakery, we are glad you came.\", 60);' ' const risky = \"Hello <b onmouseover=demo()>there</b>\";' ' console.log(\"Made-up risky reply: \" + risky);' ' console.log(\"Escaped for a page: \" + escapeHtml(risky));' ' console.log(\"Model reply escaped: \" + escapeHtml(reply));' '}' 'main();' >> risks.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the risk map
docker exec -w /home/node/project agent node risks.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| chat box -> injection | The risk and the first check for each part of the design |
| Escaped for a page | The angle brackets turned into harmless text |
| Model reply escaped | A normal reply passes through unchanged. Its words differ each run |
| Recorded sample | Printed before the escaped lines when no model answered. The script then uses the sample |
The cells students fill in are left blank in this preview.
Power, secrets and cost
Excessive agency is the risk that grows with every tool you add. Give a model the least power that does the job.
A data leak needs no attacker. A key in a prompt or a name in a log is enough. Overuse needs none either: a loop or a busy day can spend a month of budget.
A common mistake
Warning
Many people think these risks only matter when someone attacks. Most of them also happen by accident: a loop that calls the model all night, or a log that keeps a customer's email. Review every box in the design, not only the ones an attacker would reach.