Where can an AI agent be attacked?
A threat model lists what an attacker could reach and how. For an agent it matters more than for a chatbot, because an agent acts: it reads mail, calls tools and sends things out.
The Securing LLM applications module covered prompt injection and the main risks. This lesson turns them into a map of one agent, drawn before any code is written.
By the end of this lesson
Note
I can map where an agent can be attacked.
Key words
| Word | What it means | Example |
|---|---|---|
| Asset | something worth protecting, such as a customer list or a payment tool | |
| Trust boundary | the line where text or calls pass from a source you control to one you do not | |
| Untrusted content | text the agent reads that a stranger could have written | a web page |
| Way out | any route by which the agent can send data away from its owner | |
| Leg | one of the three things in the dangerous mix |
The cells students fill in are left blank in this preview.
The dangerous mix of three things
Three things together are dangerous. The agent holds private data. It reads untrusted content. It has a way out to send data away.
With two of the three, an attack stalls. Text can steer the agent, but nothing can leave, or nothing private is there to take. With all three, one hidden line can carry data out.
Worked example: three agents
| Agent | Private data | Untrusted content | Way out |
|---|---|---|---|
| Mail summariser | The inbox | Mail from strangers | None: it only shows a summary |
| Web clerk | None | Any web page | It can post to a chat |
| Inbox helper | The inbox | Mail from strangers | It can send replies |
The cells students fill in are left blank in this preview.
The Run lists legs, it attacks nothing
The first script holds four invented agents as plain data. It prints the legs of each and marks the ones with all three.
No model is called and no attack is made. A threat model is reasoning about what is possible, so a list in a script is enough to practise it.
Write the legs script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Four invented agents. Each flag says what the agent can do.' 'const agents = [' ' { name: \"mail-summariser\", data: true, untrusted: true, out: false },' ' { name: \"web-clerk\", data: false, untrusted: true, out: true },' ' { name: \"report-bot\", data: true, untrusted: false, out: true },' ' { name: \"inbox-helper\", data: true, untrusted: true, out: true },' '];' > legs.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the legs script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function legs(a) {' ' const found = [];' ' if (a.data) found.push(\"private data\");' ' if (a.untrusted) found.push(\"untrusted content\");' ' if (a.out) found.push(\"a way out\");' ' return found;' '}' 'for (const a of agents) {' ' const all = legs(a).length === 3;' ' console.log(a.name + \": \" + legs(a).join(\", \") + (all ? \" <- all three\" : \"\"));' '}' >> legs.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the legs script
docker exec -w /home/node/project agent node legs.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the script prints
| Line | What it shows |
|---|---|
| mail-summariser: private data, untrusted content | Two legs, and no way out |
| web-clerk: untrusted content, a way out | Two legs, and no private data |
| report-bot: private data, a way out | Two legs, and only trusted input |
| inbox-helper: all three legs, then <- all three | The only agent to fix first |
The cells students fill in are left blank in this preview.
Remove a leg, do not ask nicely
The fix is to take one leg away. Drop the way out, or read only trusted sources, or keep private data out of reach. Pick the one that costs the least.
A prompt that says "never leak" is not a fix. It lowers the odds on one run and gives no guarantee on the next.
A common mistake
Warning
Many people count only what the user types as the input. Every tool result, file, page and earlier message is input too. Each one is a place where a stranger may have written the text.
Write the model script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' 'const text = \"A helper reads the customer inbox, summarises it, and can post a link in a public chat room.\";' 'const messages = [' ' { role: \"system\", content: \"Answer with exactly three lines. Line 1 starts with data:, line 2 with untrusted:, line 3 with out:. After each colon write yes or no.\" },' ' { role: \"user\", content: \"Does this agent hold private data, read untrusted content, and have a way out? \" + text },' '];' '// Recorded sample, used only when no model answers.' 'const sample = \"data: yes\nuntrusted: yes\nout: yes\";' > legs2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function parse(reply) {' ' const found = {};' ' for (const line of reply.split(\"\n\")) {' ' const bits = line.split(\":\");' ' if (bits.length === 2) found[bits[0].trim().toLowerCase()] = bits[1].trim().toLowerCase().startsWith(\"yes\");' ' }' ' return found;' '}' '' 'async function main() {' ' const reply = await ask(messages, sample, 60);' ' console.log(\"Reply:\n\" + reply);' ' const f = parse(reply);' ' const keys = [\"data\", \"untrusted\", \"out\"];' ' const shape = keys.every(function (k) { return k in f; });' ' console.log(\"Shape ok: \" + shape);' ' if (shape) console.log(\"All three, by the model: \" + (f.data && f.untrusted && f.out));' ' console.log(\"The program trusts its own list of tools, not this answer.\");' '}' 'main();' >> legs2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it against the model
docker exec -w /home/node/project agent node legs2.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| Reply | Three lines that start with data, untrusted and out. The words after each colon differ each run |
| Shape ok | true when all three lines are there. false means the program must ask again or refuse |
| Recorded sample | Printed first when no model answered. The script carries on with the sample |
The cells students fill in are left blank in this preview.