A reply that arrives word by word
A streamed response is an answer that a language model sends in small pieces as it writes, instead of one block at the end. Chat apps use it so that words appear at once.
It matters because a long answer can take many seconds. With a stream the reader sees the start almost straight away. Your program has to read the pieces and join them.
By the end of this lesson
Note
I can read an answer that arrives in pieces and join it into one text.
Key words
| Word | What it means | Example |
|---|---|---|
| Stream | an answer sent in pieces as it is written | |
| Server-sent events | a way for a server to push lines to a client | |
| Data line | a stream line that starts with data: and holds JSON | |
| Delta | the new piece of text in one data line | |
| End marker | the last data line, which says the stream is over | data: [DONE] |
The cells students fill in are left blank in this preview.
A stream is a list of data lines
To ask for a stream you add "stream": true to the request body. The server then answers with server-sent events: plain text, one line at a time, with a blank line between events.
Each line that starts with data: carries one piece of JSON. The text is in choices[0].delta.content. The last line is data: [DONE].
The lines of a stream
| Line starts with | What it holds |
|---|---|
| data: {"choices":[{"delta":{"role":"assistant"}}]} | The opening: a role and no text yet |
| data: {"choices":[{"delta":{"content":"One"}}]} | A piece of text to add |
| data: {"choices":[{"delta":{},"finish_reason":"stop"}]} | The reason the reply ended |
| data: [DONE] | The end marker, not JSON |
| : keep-alive | A comment line to ignore |
The cells students fill in are left blank in this preview.
See the chunks of bytes that arrive
docker exec -w /home/node/project agent node -e 'const lab = require("./lab.js"); const event = (text) => "data: " + JSON.stringify({ choices: [{ delta: { content: text } }] }) + "\n\n"; const whole = event("One") + event(", two") + event(", three.") + "data: [DONE]\n\n"; (async () => { let chunks = []; let label = "Model stream"; try { const response = await lab.send({ stream: true, max_tokens: 30, messages: [{ role: "user", content: "Count from one to five." }] }); if (!response.ok) throw new Error("HTTP " + response.status); const decoder = new TextDecoder(); for await (const bytes of response.body) chunks.push(decoder.decode(bytes, { stream: true })); } catch (error) { label = "Recorded sample"; chunks = whole.match(/[^]{1,45}/g); } console.log(label + ", the first chunks of bytes:"); chunks.slice(0, 4).forEach((one, i) => console.log((i + 1) + " " + JSON.stringify(one))); console.log("... " + chunks.length + " chunks in all."); })();'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A network chunk is not a line
Bytes reach your program in chunks, and a chunk can stop in the middle of a line. So keep a buffer. Add each chunk to it, cut off every complete line, and hold the rest for the next chunk.
Only a complete line is safe to read as JSON. Then add each piece of text to the answer.
Write stream.js, a reader of whole lines
docker exec -w /home/node/project agent node -e 'require("fs").writeFileSync("stream.js", ["// stream.js: read a streamed answer as complete lines, however the chunks are cut.", "async function* linesOf(response) {", " let held = \"\";", " const decoder = new TextDecoder();", " for await (const bytes of response.body) {", " held += decoder.decode(bytes, { stream: true });", " const parts = held.split(\"\\n\");", " held = parts.pop();", " yield* parts;", " }", " if (held !== \"\") yield held;", "}", "", "module.exports = { linesOf };"].join("\n") + "\n")'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Add piecesOf, which gives the text
docker exec -w /home/node/project agent node -e 'require("fs").appendFileSync("stream.js", ["", "// Part 2: send a streamed call and give back the pieces of text, one at a time.", "async function* piecesOf(body) {", " const response = await require(\"./lab.js\").send({ ...body, stream: true });", " if (!response.ok) throw new Error(\"HTTP \" + response.status);", " for await (const line of linesOf(response)) {", " if (!line.startsWith(\"data: \") || line === \"data: [DONE]\") continue;", " const piece = JSON.parse(line.slice(6)).choices[0].delta.content;", " if (piece) yield piece;", " }", "}", "", "module.exports = { linesOf, piecesOf };"].join("\n") + "\n")'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think each chunk is one event. It is not. A chunk may hold three events or half of one. Your program works when it gives the same text however the bytes are cut. Buffer first, then read lines.
Print and join the pieces
docker exec -w /home/node/project agent node -e 'const { piecesOf } = require("./stream.js"); const sample = ["One", ", two", ", three", ", four", ", five."]; (async () => { const started = Date.now(); let first = null, count = 0, text = "", label = "Model answer"; try { const asking = { max_tokens: 40, messages: [{ role: "user", content: "Count from one to five." }] }; for await (const piece of piecesOf(asking)) { if (first === null) first = Date.now() - started; text += piece; count++; process.stdout.write(piece); } } catch (error) { label = "Recorded sample"; text = sample.join(""); count = sample.length; first = null; process.stdout.write(text); } console.log("\n" + label + ": " + count + " pieces, " + text.length + " characters."); if (first !== null) console.log("The first piece came after " + first + " ms and all of it after " + (Date.now() - started) + " ms."); })();'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: from lines to text
| Data lines in order | Pieces added | Text so far |
|---|---|---|
| Opening, role only | None | Empty |
| content "One" | One | One |
| content ", two" | , two | One, two |
| finish_reason "stop" | None | One, two |
| data: [DONE] | None | One, two |
The cells students fill in are left blank in this preview.
Going further: time the first piece
Run the assemble script three times and write down the time to the first piece and to the whole answer.
Then say which of the two a reader notices more, and why a stream helps most when the answer is long.