Show the answer as it arrives
A streaming chat page shows a chatbot's answer a few words at a time, as the server sends them. It matters because a long answer can take many seconds, and a blank page makes people think the site is broken.
The server sends small pieces. The page adds each piece to the screen. The lab terminal has no browser, so here you build the page and the stream as text. On your own machine you would open it in a browser.
By the end of this lesson
Note
I can show a streamed answer in a web page as it arrives.
Key words
| Word | What it means |
|---|---|
| Stream | an answer sent in small pieces |
| Server-sent events | a way for a server to push lines to a page |
| Handler | the function that answers one request |
| Escape | to change characters so they show as text |
| Piece | one small part of a streamed answer |
The cells students fill in are left blank in this preview.
A server can send an answer in pieces
With server-sent events the server keeps the connection open and writes lines. Each event starts with data: , holds one piece as JSON and ends with a blank line. The type of the response is text/event-stream.
A last event, named done, tells the page the answer is finished.
Worked example: the lines of a stream
| Line sent | What the page does |
|---|---|
| data: {"piece": "Hello"} and a blank line | Adds Hello to the screen |
| data: {"piece": " and welcome"} and a blank line | Adds the next words after it |
| event: done, then data: end and a blank line | Closes the connection |
The cells students fill in are left blank in this preview.
The page appends and can stop
The page holds a small script. It opens the stream, and for each event it adds the piece to a paragraph. A stop button closes the stream, so no more pieces arrive.
Adding a piece with append and a string adds plain text. It never adds markup.
Escape model text before it becomes markup
A model can repeat what it was given, including <script>. If that text is put into a page as HTML, the browser may run it.
Escaping turns < into < and the other special characters into safe codes. Escape any model text that goes into HTML. Do & first, or you escape your own codes.
A common mistake
Warning
Many people think model text is safe because they wrote the prompt. It is not. The model can echo a user's words or a source's words. Treat every reply as untrusted: add it as text, or escape it first.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page and stream, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' 'function quoted(text) { return JSON.stringify(text); }' 'function escapeHtml(text) {' ' return text.replace(/&/g, \"&\").replace(/</g, \"<\").replace(/>/g, \">\")' ' .replace(/\"/g, \""\").replace(/\x27/g, \"'\");' '}' > ui4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page and stream, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function pageHtml() {' ' const script = [' ' \"const out = document.getElementById(\" + quoted(\"answer\") + \");\",' ' \"const source = new EventSource(\" + quoted(\"/stream\") + \");\",' ' \"source.onmessage = function (e) { out.append(JSON.parse(e.data).piece); };\",' ' \"source.addEventListener(\" + quoted(\"done\") + \", function () { source.close(); });\",' ' \"document.getElementById(\" + quoted(\"stop\") + \").onclick = function () { source.close(); };\",' ' ].join(\"\n\");' ' return \"<html><body><p id=answer></p><button id=stop>Stop</button><script>\" + script + \"</script></body></html>\";' '}' >> ui4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page and stream, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function sseLine(piece) { return \"data: \" + JSON.stringify({ piece: piece }) + \"\n\n\"; }' 'function streamBody(pieces, stopAfter) {' ' let body = \"\";' ' pieces.slice(0, stopAfter).forEach(function (piece) { body = body + sseLine(piece); });' ' return stopAfter >= pieces.length ? body + \"event: done\ndata: end\n\n\" : body;' '}' >> ui4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page and stream, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function handle(request, pieces) {' ' if (request.url === \"/\") return { status: 200, type: \"text/html\", body: pageHtml() };' ' if (request.url === \"/stream\") {' ' return { status: 200, type: \"text/event-stream\", body: streamBody(pieces, request.stopAfter || 99) };' ' }' ' return { status: 404, type: \"text/plain\", body: \"Not found\" };' '}' >> ui4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page and stream, part 5
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function show(pieces) {' ' const home = handle({ url: \"/\" }, pieces);' ' console.log(\"GET / ->\", home.status, home.type, home.body.length + \" characters\");' ' const live = handle({ url: \"/stream\" }, pieces);' ' console.log(\"GET /stream ->\", live.status, live.type);' ' console.log(live.body.split(\"\n\n\").join(\" | \"));' ' const stopped = handle({ url: \"/stream\", stopAfter: 2 }, pieces);' ' console.log(\"Stop pressed after 2 pieces:\", stopped.body.split(\"\n\n\").length - 1, \"pieces sent, no done line\");' ' console.log(\"Escaped:\", escapeHtml(\"<b>hi</b> & bye\"));' '}' >> ui4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page and stream, part 6
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' // Recorded sample, used only when no model answers.' ' const sample = \"Hello and welcome aboard, where would you like to go?\";' ' const reply = await ask([{ role: \"user\", content: \"Greet a bus passenger in one short sentence.\" }], sample, 40);' ' show(reply.split(\" \").map(function (word, at) { return at === 0 ? word : \" \" + word; }));' '}' 'main();' >> ui4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Call the handler
docker exec -w /home/node/project agent node ui4.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it may show |
|---|---|
| GET / | 200 text/html and the size of the page |
| GET /stream | 200 text/event-stream, then the pieces of one reply with | between them |
| Stop pressed | 2 pieces sent, no done line |
| Escaped | <b>hi</b> & bye, the same every time |
The cells students fill in are left blank in this preview.