A slow call must not freeze your program
A call to a model can take several seconds. Your program should not freeze while it waits. Async code lets it wait without blocking.
Streaming goes one step further. The answer arrives in pieces, so the user can start reading at once.
By the end of this lesson
Note
I can say why a slow call is awaited and how a streamed answer arrives.
Key words
| Word | What it means |
|---|---|
| Promise | a stand-in for a result that has not arrived yet |
| Await | to pause a function until a promise has its result |
| Async function | a function that is allowed to await |
| Stream | an answer sent in pieces as it is made |
| Chunk | one piece of a streamed answer |
The cells students fill in are left blank in this preview.
A slow call returns a promise
When you start a slow job, the call returns at once with a promise. The result is not there yet. Your program carries on with the next line.
When the result arrives, the promise delivers it. Order matters: lines after the call may run before the result.
See the order things happen
docker exec -w /home/node/project agent node -e "const wait=(ms,t)=>new Promise(r=>setTimeout(()=>r(t),ms));console.log('1 start');wait(300,'3 the slow call is done').then(console.log);console.log('2 the program carries on')"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Await pauses one function, not the program
Inside an async function, await pauses that function until the promise has its result. The rest of the program is not blocked.
Two slow calls that do not depend on each other can start together with Promise.all. That takes about as long as the slower one, not both added up.
Await one by one, then together
docker exec -w /home/node/project agent node -e "const wait=(ms,t)=>new Promise(r=>setTimeout(()=>r(t),ms));const tick=(t0)=>Math.round((Date.now()-t0)/100)*100;(async()=>{let t0=Date.now();await wait(200,'a');await wait(200,'b');console.log('one after the other: about '+tick(t0)+' ms');t0=Date.now();await Promise.all([wait(200,'a'),wait(200,'b')]);console.log('together: about '+tick(t0)+' ms')})()"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A streamed answer arrives in chunks
If you ask for stream: true, the service sends the answer as it makes it. Most model APIs use server-sent events: lines that start with data:, each holding a small piece of text.
The last line is data: [DONE]. Join the pieces in order to get the whole answer.
Worked example: a stream, line by line
| Line received | What it gives |
|---|---|
| data: {"choices":[{"delta":{"content":"Hel"}}]} | Hel |
| data: {"choices":[{"delta":{"content":"lo"}}]} | lo |
| data: [DONE] | The end, no text |
| Joined | Hello |
The cells students fill in are left blank in this preview.
Read a streamed answer
docker exec -w /home/node/project agent node -e "const u=(process.env.AI_URL||'http://ai:4000')+'/v1/chat/completions';(async()=>{try{const r=await fetch(u,{method:'POST',headers:{'content-type':'application/json',authorization:'Bearer lab'},body:JSON.stringify({model:'lab',messages:[{role:'user',content:'Count from one to five in words.'}],max_tokens:40,stream:true})});if(r.ok===false)throw new Error(r.status);const d=new TextDecoder();let b='',n=0,t='';for await(const p of r.body){b+=d.decode(p,{stream:true});const L=b.split(String.fromCharCode(10));b=L.pop();for(const l of L){const x=l.slice(6);if(l.startsWith('data: ')&&(x==='[DONE]')===false){const c=JSON.parse(x).choices?.[0]?.delta?.content;if(c){n++;t+=c}}}}console.log('Chunks with text: '+n);console.log('Joined text: '+t)}catch(e){console.log('No stream ('+e.message+'). Recorded sample:');console.log('Chunks with text: 5');console.log('Joined text: one, two, three, four, five')}})()"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think streaming makes the model faster. The whole answer takes about as long. You see the first words sooner, which feels faster. A stream can also stop half way, so check that the end marker arrived.