Step by step prompting asks for working first
Step by step prompting means asking a model to write its working before its answer. For a sum or a plan, the steps can make the answer more likely to be right.
Steps cost words, and words cost money and time. So the skill is knowing when to ask for them and checking the answer when you do.
By the end of this lesson
Note
I can ask for steps when steps help and not when they do not.
Key words
| Word | What it means |
|---|---|
| Step | one small piece of working |
| Working | the steps that lead to an answer |
| Answer line | the last line that holds only the answer |
| Check | a test that the answer is right |
| Cost | the tokens and time a reply uses |
| Recorded | a reply saved earlier, not made now |
The cells students fill in are left blank in this preview.
Why can steps help?
A model writes one piece of text at a time, and each piece can use what it already wrote. When the working is on the page, the next step can use it. A total of 30 is easier to reach through 2, then 6, then 30.
This helps with sums, plans and rules with several parts. It helps little with a plain lookup, such as the capital of a country.
When to ask for steps
| Question | Steps? | Why |
|---|---|---|
| Total cost of 15 bus rides at 2 pounds | Yes | Two connected sums |
| Which of these rules did the user break? | Yes | Several parts to check |
| Rewrite this line more politely | No | One move, nothing to work out |
| Is this comment good or bad? | No | A label is the whole answer |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the step splitter, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const QUESTION = \"A bus ride costs 2 pounds. Sam rides 3 times a day for 5 days. What is the total cost?\";' 'const RECORDED = \"Step 1: A ride costs 2 pounds.\nStep 2: 3 rides a day cost 6 pounds.\nStep 3: 5 days cost 30 pounds.\nAnswer: 30 pounds\";' > steps3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the step splitter, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function split(reply) {' ' const steps = [];' ' let answer = \"\";' ' for (const line of reply.split(\"\n\")) {' ' if (line.startsWith(\"Step \")) steps.push(line.slice(line.indexOf(\":\") + 1).trim());' ' if (line.startsWith(\"Answer:\")) answer = line.slice(7).trim();' ' }' ' return { steps: steps, answer: answer };' '}' 'function show(label, reply) {' ' const r = split(reply);' ' console.log(label + \": \" + r.steps.length + \" steps, answer \" + (r.answer || \"none\"));' '}' >> steps3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the step splitter, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' show(\"recorded\", RECORDED);' ' const form = \" Work in short steps that start with Step 1: and so on. End with one line that starts with Answer:\";' ' const reply = await ask([{ role: \"user\", content: QUESTION + form }], RECORDED, 150);' ' show(\"model\", reply.text);' '}' 'main();' >> steps3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run and split a reply
docker exec -w /home/node/project agent node steps3.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
The first line splits a recorded reply: 3 steps, answer 30 pounds. The second line does the same for the lab's model. With no model it prints a Recorded sample and the second line is the same as the first.
A model may not keep your format. If its line says answer none, the model did not write a line starting Answer:. Your code must expect that.
A common mistake
Warning
Many people think that if the steps look sensible, the answer is right. A reply can show tidy steps and then write a different answer. Always compare the answer line with the last step, or with a value you know.