Why model calls go through a gateway
An LLM gateway is one service that every model call passes through. It holds the provider keys, applies limits, logs the call and picks the model. It matters because every product that calls a provider directly needs its own copy of each rule.
A company has three products that call a model. One leaks a key in a build log. One has no limit and spends a month of budget in a night. A gateway turns both into one fix.
By the end of this lesson
Note
I can say what a gateway in front of model calls is for.
Key words
| Word | What it means |
|---|---|
| Gateway | one service all model calls pass through |
| Client key | a key a team uses to reach the gateway |
| Rate limit | a cap on requests in a time window |
| Routing | choosing which model serves a request |
| Request log | a record of each call, without its text |
The cells students fill in are left blank in this preview.
A gateway does four jobs
Each job is a rule applied before or after the call. Keys: callers hold a client key, and only the gateway holds the provider key. Limits: a count per team per minute. Routing: a rule picks the model. Logging: one record per call.
Because every call passes one function, a change to a rule is a change in one place.
Worked example: five requests, limit two
| Request | Key | Words | Outcome |
|---|---|---|---|
| 1 | key-red | 40 | ok, small-model |
| 2 | key-red | 500 | ok, large-model |
| 3 | key-red | 20 | limited: red already has two |
| 4 | key-zzz | 10 | denied: no such key |
| 5 | key-blue | 80 | ok, small-model |
The cells students fill in are left blank in this preview.
What a gateway may log
Log the id, the team, the status, the model, the size and the time. Do not log the prompt text. Prompts hold personal data, and a log is read by more people than the feature is.
The gateway is also a single point of failure and an added hop. Give it a timeout, run more than one copy and keep its rules small.
A common mistake
Warning
Many people think a gateway makes calls safe because it checks keys. It checks who is calling, not what is asked. A valid key can still send a harmful prompt. Limits and logs bound the damage; they do not judge the text.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the gateway, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const KEYS = { \"key-red\": \"red\", \"key-blue\": \"blue\" }; // invented keys' 'const LIMIT = 2; // accepted requests a team may make' 'const counts = {};' 'const log = [];' > gateway.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the gateway, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function choose(request) {' ' return request.words > 300 ? \"large-model\" : \"small-model\";' '}' 'function gateway(request) {' ' const team = KEYS[request.key];' ' let status = \"ok\";' ' if (team === undefined) status = \"denied\";' ' else if ((counts[team] || 0) >= LIMIT) status = \"limited\";' ' if (status === \"ok\") counts[team] = (counts[team] || 0) + 1;' ' const model = status === \"ok\" ? choose(request) : \"none\";' ' log.push({ id: request.id, team: team || \"unknown\", status: status, model: model });' ' return status;' '}' >> gateway.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the gateway, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const requests = [' ' { id: 1, key: \"key-red\", words: 40 },' ' { id: 2, key: \"key-red\", words: 500 },' ' { id: 3, key: \"key-red\", words: 20 },' ' { id: 4, key: \"key-zzz\", words: 10 },' ' { id: 5, key: \"key-blue\", words: 80 },' '];' 'requests.forEach(function (r) { gateway(r); });' 'log.forEach(function (row) { console.log(row.id, row.team, row.status, row.model); });' >> gateway.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the gateway, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const sample = \"A gateway keeps keys, limits and logs in one place.\";' ' const messages = [{ role: \"user\", content: \"In one short sentence: why put a gateway before model calls?\" }];' ' console.log(\"Live call for request 1:\", await ask(messages, sample, 40));' '}' 'main();' >> gateway.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the gateway
docker exec -w /home/node/project agent node gateway.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A stand-in, not a real gateway
This gateway is a function over a list. The lab has no network of services, so the limit lives in a variable. A real gateway is a service with shared storage for its counts, so every copy sees the same numbers.
The platform already acts as your gateway here: it adds your token and picks the model for the one live call.
One possible output
| Line | What it shows |
|---|---|
| 1 red ok small-model | Accepted and sent to the small model |
| 3 red limited none | Refused, nothing sent to a model |
| 4 unknown denied none | Refused, the team is unknown |
| Live call | One short sentence from a model, or a labelled Recorded sample |
The cells students fill in are left blank in this preview.