Capstone: an agent with safe limits

An AI agent is a program where a model chooses actions, one step at a time. Because an agent acts, its limits decide how much harm one mistake can do.

Say an IT team wants one that reads a ticket, finds the user and unlocks their account. It could unlock the wrong person, or loop for an hour. In this capstone you plan its limits and review a recorded run.

The model may ask for any action it likes. Your code decides what actually runs.

An allowlist is the short list of tools the agent may use. Your program checks every request against it before running anything. Anything not on the list is blocked, however reasonable it sounds. The program below checks a recorded sample of requests.

// The agent may only use these tools.
const allowed = ["read_ticket", "look_up_user", "unlock_account", "reply"];

// A recorded sample: the tools the model asked for, in order.
const asked = ["read_ticket", "look_up_user", "delete_user", "unlock_account", "reply"];

for (const tool of asked) {
  if (allowed.includes(tool)) {
    console.log("run   ", tool);
  } else {
    console.log("BLOCK ", tool);
  }
}

// Add "reset_password" to the asked list, then press Run.
// Is it run or blocked? Should it be on the allowlist?

A budget stops an agent that loops. Five steps is plenty to unlock one account. Past that, the agent stops and hands the ticket to a person.

A human check is for actions that are hard to undo. Unlocking an account changes who can get in, so a person approves it first. Reading a ticket is safe, so it runs at once.

Every step goes in a run log: the tool, its input and its result. After a test run, you read the log line by line.

Look for three things. A tool that should not be there. Steps past the budget. A risky step with no approval. The program prints a recorded run with notes beside each step.

# A recorded run of the help desk agent. Budget: 5 steps.
run = ["read_ticket", "look_up_user", "unlock_account",
       "look_up_user", "unlock_account", "reply"]
BUDGET = 5
RISKY = ["unlock_account"]

step = 1
for tool in run:
    note = ""
    if tool in RISKY:
        note = "needs a person to approve"
    if step > BUDGET:
        note = "over budget"
    print(step, tool, note)
    step = step + 1

# Change BUDGET to 6 and run it again.