Who remembers the conversation?
A school library has a chat window on its website. You type your name, then ask a question, and it still knows you. The language model behind it remembers nothing. A small program in the middle does: it keeps the conversation and protects the people using it. In this lab you test that program, against a stand-in model that always gives the same answer.
By the end of this lesson
Note
I can explain what a chat service adds around a language model.
Key words
- Session: one conversation, kept under its own name.
- Context budget: how many tokens of history are sent each time.
- System prompt: rules the service sends first, which the user cannot change.
- Persona: the name and manner the helper is given.
- Moderation: checking words before they are sent or shown.
- Rate limit: how many messages one user may send in a minute.
Starter: what you already know
Note
Answer in your head. 1. Which message role sets the rules? 2. Does a model remember the last request? 3. What should a program do when a model says it is busy? Answers: system; no, it knows only what each request sends; wait, then try again.
Before you start
Note
Press Start the lab on the bar at the top of this lesson and wait until it says it is running. There are two containers. model is the stand-in. chat is the chat service at http://chat:8000/chat, and the terminal's curl reaches it. docker exec chat runs a command inside it. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.
A session keeps the conversation
Each message names a user and a session. In session s1 you say "My name is Ana.", then ask your name: the reply is "Your name is Ana." The service sent the earlier messages again. In session s2 the same question gets "I do not know your name." Sessions never mix, so one person's chat never reaches another.
curl -s -w "\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u1","session":"s1","message":"My name is Ana."}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
curl -s -w "\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u1","session":"s1","message":"What is my name?"}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
curl -s -w "\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u2","session":"s2","message":"What is my name?"}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
History has a budget, and the system prompt comes first
Every token sent costs money and time, so the service sends only the newest history that fits. The script asks four questions twice. With a budget of 120 tokens it sends 85 and still knows Ana. With 40 it sends 37, the message with the name is left out, and the reply is "I do not know your name." The system prompt and the persona always go first: the books persona refuses a sum.
docker exec chat python3 trim_demo.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
curl -s -w "\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u3","session":"s3","persona":"books","message":"What is 6 times 7?"}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
Check
Tip
A long chat suddenly forgets the user's name. What happened? Answer: the message with the name no longer fitted the context budget, so it was not sent. Raise the budget, or keep a short summary of what matters.
Words are checked on the way in and on the way out
"You are useless." is never sent: moderated is input and tries is 0. The model's reply about the librarian held an email address and a phone number, and both became [removed]. A stream comes in pieces, so the service waits for each whole sentence, checks it, then sends it: four sentences, then [DONE].
curl -s -w "\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u4","session":"s4","message":"You are useless."}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
curl -s -w "\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u4","session":"s4","message":"How do I contact the librarian?"}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
curl -s -N http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u4","session":"s5","stream":true,"message":"What are the library rules?"}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
A busy model is retried, and fixed transcripts are tests
The stand-in refuses #busy twice. The service waits half a second, then one second, and answers on try 3. #broken always fails, so after three tries the user gets an apology and status 503, never an error. Each user may send 10 messages a minute. The log keeps counts and hashed ids, never words. The tests replay fixed conversations: 9 checks, 0 failed.
curl -s -w "\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u5","session":"s6","message":"What are the library hours? #busy"}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
curl -s -w "\n%{http_code}\n" http://chat:8000/chat -H "Content-Type: application/json" -d '{"user":"u5","session":"s7","message":"What are the library hours? #broken"}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec chat python3 test_transcripts.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
A common mistake
Warning
Many people think a system prompt is enough to keep a chatbot safe. It is not: a model can still say something it should not. Check what goes in and what comes out in the service, where the rules always run.
Exit check
Tip
Answer without looking back. 1. Who keeps the conversation, the model or the service? 2. Why trim history to a budget? 3. Why check a reply before showing it? 4. What may a chat log keep? Answers: the service; tokens cost money and time; a model can say things it should not; counts and hashed ids, never words.
For teachers
Tip
This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is two services of one Python image: a stand-in model that answers by fixed rules, and a chat service written with Python's standard library. No real model is called, so it costs nothing and every output is the same. Key learning points: 1. The service keeps each conversation, not the model. 2. History is trimmed to a token budget. 3. Words are checked in and out. 4. A busy model is retried and a broken one gets an apology. Common mistake: trusting the system prompt alone. Equipment: the lab and a browser. Tip: ask students to add a refused word to chatbot.py and a transcript that proves it, so they change the service and test it. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.