A model guesses what comes next
A language model is a program that writes text by guessing what comes next. Your phone does a tiny version when it suggests the next word of a message.
It matters because every feature you build on a model inherits that guess. Once you see how it works, its odd behaviour makes sense.
By the end of this lesson
Note
I can explain how a language model produces text, without the maths.
Key words
| Word | What it means |
|---|---|
| Model | a program that learned patterns from a huge amount of text |
| Prompt | the text you send to a model |
| Token | a small piece of text, often a word or part of one |
| Context | all the text the model can see for one answer |
| Temperature | a setting that makes the guess more or less varied |
The cells students fill in are left blank in this preview.
A model writes one token at a time
A model does not plan a whole answer. It reads your prompt, guesses the next token, adds it and guesses again.
It repeats this until the answer ends. Every token is chosen using all the text so far, the context.
Worked example: the next word
| The text so far | Next word | Seen out of 10 |
|---|---|---|
| The cat sat on the | mat | 6 |
| The cat sat on the | sofa | 3 |
| The cat sat on the | roof | 1 |
The cells students fill in are left blank in this preview.
Ask for the next word
docker exec -w /home/node/project agent node -e "const u=(process.env.AI_URL||'http://ai:4000')+'/v1/chat/completions';const post=(m,t,x)=>fetch(u,{method:'POST',headers:{'content-type':'application/json',authorization:'Bearer lab'},body:JSON.stringify({model:'lab',messages:[{role:'user',content:m}],max_tokens:t,temperature:x})}).then(r=>r.json());post('Finish the sentence with one word and nothing else: The cat sat on the',12,0.7).then(j=>console.log('Model: '+j.choices[0].message.content)).catch(e=>console.log('Recorded sample (no model answered): mat'))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The same prompt can give different answers
The model gives each possible token a chance and then picks one. Temperature is the setting that decides how adventurous the pick is. A low temperature picks the likely token almost every time. A high one picks unusual tokens more often.
So a model is not a calculator. Ask twice and you may get two answers.
Ask the same prompt twice
docker exec -w /home/node/project agent node -e "const u=(process.env.AI_URL||'http://ai:4000')+'/v1/chat/completions';const post=(m,t,x)=>fetch(u,{method:'POST',headers:{'content-type':'application/json',authorization:'Bearer lab'},body:JSON.stringify({model:'lab',messages:[{role:'user',content:m}],max_tokens:t,temperature:x})}).then(r=>r.json());const s=['mat','sofa'];(async()=>{for(let i=0;i<2;i++){let t;try{const j=await post('Finish the sentence with one word and nothing else: The cat sat on the',12,1);t='Model: '+j.choices[0].message.content}catch(e){t='Recorded sample (no model answered): '+s[i]}console.log('Try '+(i+1)+': '+t)}})()"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Likely is not the same as true
The model learned from text, so it is good at what sounds right. It has no way to look a fact up. If a wrong sentence looks like a likely sentence, it can write the wrong one in a calm, sure voice.
Your job as a developer is to check what matters.
Ask about something it cannot know
docker exec -w /home/node/project agent node -e "const u=(process.env.AI_URL||'http://ai:4000')+'/v1/chat/completions';const post=(m,t,x)=>fetch(u,{method:'POST',headers:{'content-type':'application/json',authorization:'Bearer lab'},body:JSON.stringify({model:'lab',messages:[{role:'user',content:m}],max_tokens:t,temperature:x})}).then(r=>r.json());post('What is the name of my cat? Answer in one short sentence.',40,0.7).then(j=>console.log('Model: '+j.choices[0].message.content)).catch(e=>console.log('Recorded sample (no model answered): I do not know the name of your cat.'))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a model looks facts up, like a search engine. It does not. It continues text with likely tokens. Trust an answer only after you check it against something else.
Going further: a tiny model of your own
The assignment is a tiny language model: a table of counts and a rule for picking. A real model is the same idea. Its table is far bigger, it is built from far more text and it looks at much more of the context.
Try adding a few words and counts of your own to the table and see what text it writes.