Three settings that change an answer
Temperature, a token limit and stop text are the three settings of a model call that change the answer most. Each one goes in the request body beside the messages.
You need them because a defaults-only call can be too random for a data job and too long for a screen. Knowing what each does lets you choose on purpose.
By the end of this lesson
Note
I can say what temperature, a token limit and stop text change in a model call.
Key words
| Word | What it means | Example |
|---|---|---|
| Temperature | a setting for how varied the answers are | 0 |
| Token | a small piece of text a model reads and writes | |
| Max tokens | the most tokens the model may write in a reply | |
| Stop text | text that ends the reply when the model reaches it | |
| Finish reason | why the reply ended | length |
The cells students fill in are left blank in this preview.
Temperature sets how varied the answers are
A model picks each next token from a list of likely ones. A low temperature takes the most likely one almost every time. A high temperature lets less likely ones in.
So choose a low value, near 0, for a job with one right answer, such as pulling a date out of text. Choose a higher value for ideas and names.
Worked example: temperature for three jobs
| Job | Temperature | Why |
|---|---|---|
| Pull an order number out of an email | 0 | There is one right answer |
| Answer a customer question | 0.7 | Natural, but not wild |
| Suggest ten names for a shop | 1.2 | You want variety |
The cells students fill in are left blank in this preview.
Ask three times at two temperatures
docker exec -w /home/node/project agent node -e 'const lab = require("./lab.js"); const question = [{ role: "user", content: "Name a pet goldfish. Answer with one word." }]; const samples = { 0: ["Bubbles", "Bubbles", "Bubbles"], 1.2: ["Finn", "Captain Splash", "Bubbles"] }; (async () => { for (const temperature of [0, 1.2]) { const found = []; let label = ""; for (let i = 0; i < 3; i++) { const got = await lab.ask({ max_tokens: 8, temperature: temperature, messages: question }, samples[temperature][i]); found.push(got.text.trim()); label = got.label; } console.log("temperature " + temperature + " (" + label + "): " + found.join(" | ")); } })();'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Max tokens is a hard cap
A token is a small piece of text, often a short word or part of a long one. max_tokens is the most tokens the model may write. When it is reached the reply stops, even in the middle of a sentence.
The reply carries a finish reason. The word stop means the model finished. The word length means the cap cut it off.
Cut a reply short on purpose
docker exec -w /home/node/project agent node -e 'const lab = require("./lab.js"); const question = [{ role: "user", content: "Explain what an API is in two sentences." }]; const sample = { short: "An API is a way for one program to ask", long: "An API is a set of rules that lets one program ask another program for something. The caller sends a request and gets an answer back." }; (async () => { for (const limit of [12, 80]) { const got = await lab.ask({ max_tokens: limit, messages: question }, limit === 12 ? sample.short : sample.long); const why = got.data ? got.data.choices[0].finish_reason : (limit === 12 ? "length" : "stop"); console.log("max_tokens " + limit + " (" + got.label + "), finish_reason " + why + ":"); console.log(" " + got.text.trim()); } })();'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a small max_tokens makes the model write a shorter answer. It does not. It cuts the answer off, often mid-sentence. To get a shorter answer, ask for one in the prompt. Keep the cap as a safety limit above what you expect.
Stop text ends a reply early
A stop text is text you choose. When the model is about to write it, the reply ends instead, and the stop text itself is left out.
Use it to keep a reply to one line or one list item. Most model APIs let you give a few stop texts at once.
Stop a list before item three
docker exec -w /home/node/project agent node -e 'const lab = require("./lab.js"); const question = [{ role: "user", content: "Write the numbers one to five in words, one per line, each line starting 1. 2. 3. and so on." }]; (async () => { const got = await lab.ask({ max_tokens: 60, stop: ["3."], messages: question }, "1. one\n2. two\n"); console.log("Reply (" + got.label + "): " + JSON.stringify(got.text)); console.log("Does it contain 3. ? " + got.text.includes("3.")); })();'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Going further: try your own settings
Change the question in the length Run to one of your own and try a max_tokens of 5, 30 and 200. For each, write down the finish reason.
Then say which value you would put in a real program. Say why a cap that is too small costs more than one that is too big.