Why a chatbot forgets your name
A chatbot keeps the messages of a conversation and sends them again with every new question. It matters because a call to a language model has no memory of its own.
A customer tells a bus company's chatbot about a lost ticket. Then the customer asks "and for my brother?" The bot can only follow if the earlier messages travel with the question.
By the end of this lesson
Note
I can keep a conversation between requests.
Key words
| Word | What it means | Example |
|---|---|---|
| Session | one person's conversation with the chatbot | |
| Session id | a label that picks out one session | ana |
| State | what the program remembers between requests | |
| Store | the place where all sessions are kept | |
| Message list | the messages sent with every call |
The cells students fill in are left blank in this preview.
A model call starts from nothing
Each call to a model starts fresh. The model knows only the messages you send in that call.
To make a conversation, your program keeps every message. It sends the whole list each time, with the new question at the end. That list is the state of the chat.
Worked example: three requests
| Request | Messages sent | What the model can see |
|---|---|---|
| 1 | system, user | Only the rule and "My name is Ana." |
| 2 | system, user, assistant, user | The first exchange and "What is my name?" |
| 3 | system, user, assistant, user, assistant, user | Everything so far, and the newest question |
The cells students fill in are left blank in this preview.
Each person gets their own session
A chatbot talks to many people at once. So the store is keyed by session id: one list for each conversation. A new id starts a new list. A known id adds to its list.
The store can be a variable while the program runs. A small JSON file in the project folder keeps it after the program stops.
Who owns the state?
Your program owns the state, not the model. So you decide what is kept and for how long.
Do not store passwords, card numbers or anything the chatbot does not need. Give people a way to delete a session. Deleting a session removes the whole list.
A common mistake
Warning
Many people think the model remembers who is talking. It does not. If two people share one message list, each sees the other's words. Keep one list for each session id.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the session store, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const { ask } = require(\"./model\");' 'const FILE = \"sessions.json\"; // the store lives in the project folder' 'const RULE = \"You are a helpful assistant. Answer in one short sentence.\";' '' 'function load() {' ' try { return JSON.parse(fs.readFileSync(FILE, \"utf8\")); } catch (error) { return {}; }' '}' 'function save(store) { fs.writeFileSync(FILE, JSON.stringify(store, null, 1)); }' > chat1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the session store, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function say(store, id, text, samples) {' ' if (store[id] === undefined) store[id] = [{ role: \"system\", content: RULE }];' ' store[id].push({ role: \"user\", content: text });' ' const answer = await ask(store[id], samples, 40);' ' store[id].push({ role: \"assistant\", content: answer });' ' save(store);' ' return answer;' '}' >> chat1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the session store, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const store = {};' ' // Recorded samples, used only when no model answers.' ' const anaSamples = [\"Nice to meet you, Ana.\", \"Your name is Ana.\"];' ' const benSamples = [\"I do not know your name yet.\"];' ' console.log(\"Ana:\", await say(store, \"ana\", \"My name is Ana. Remember it.\", anaSamples));' ' console.log(\"Ana:\", await say(store, \"ana\", \"What is my name?\", anaSamples));' ' console.log(\"Ben:\", await say(store, \"ben\", \"What is my name?\", benSamples));' ' const saved = load();' ' console.log(\"Saved: ana has \" + saved.ana.length + \" messages, ben has \" + saved.ben.length);' '}' 'main();' >> chat1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run two conversations
docker exec -w /home/node/project agent node chat1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| Ana (first) | A short greeting that may use the name Ana |
| Ana (second) | An answer that may say the name, because it was sent the earlier messages |
| Ben | An answer that cannot know Ana's name, because Ben has his own list |
| Saved | ana has 5 messages, ben has 3 |
The cells students fill in are left blank in this preview.