Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

LLM API Basics

The whole lesson is free and interactive. Read it here, or press Start lab to run it for real in a live lab, with no account needed.

Container lab10 min read8 commands

What does an app send to a chatbot?

A homework helper or a shop's chat window is a program. It sends your words to a language model and shows what comes back. In this lab you write that program's side: the request, the answer and what goes wrong. The model here is a stand-in. It answers like a real one but always gives the same answer, so you can check every result.

By the end of this lesson

Note

I can send a request to a language model API and read every part of its answer.

Key words

  • API: a way for one program to ask another for something.
  • Message: one turn of a conversation, with a role and some text.
  • Role: who a message is from: system, user or assistant.
  • Token: a small piece of text a model reads or writes.
  • Stream: an answer sent in pieces as it is made.

Starter: what you already know

Note

Answer in your head. 1. What does an HTTP status of 200 mean? 2. In JSON, what do the curly brackets {} hold? 3. If a website is busy, is it better to try again at once or after a pause? Answers: it worked; names and values; after a pause.

Before you start

Note

Press Start the lab on the bar at the top of this lesson and wait until it says it is running. There are two containers. model is the stand-in, answering at http://model:4000/v1, and the terminal's curl reaches it directly. tools has the lab's Python, and docker exec tools runs one command inside it. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.

A request is JSON, and so is the answer

The first command asks which models there are: one, called lab. The second sends one user message. In the answer, choices holds the reply, "The capital of France is Paris.", and finish_reason is stop: it ended by itself. usage counts 15 tokens in and 10 out.

Command
curl -s -w "\n" http://model:4000/v1/models

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
curl -s -w "\n" http://model:4000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"lab","messages":[{"role":"user","content":"What is the capital of France?"}]}'

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

The system message sets the rules, and nothing is remembered

The same question three times. A system message saying "Answer in capital letters." gives the reply in capitals, and "Only answer questions about cooking." gives a polite refusal. Then the user says "My name is Sam." The name is known only when the earlier messages are sent again. The model keeps nothing between requests: your program does.

Command
docker exec tools python3 steps/roles.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Tokens are what you are charged for

The stand-in splits "The lighthouse keeper counted ships." into 11 tokens. A whole story used 14 tokens in and 66 out. With max_tokens 12 the reply stops mid-word and finish_reason is length. At made-up prices of 50 pence a million tokens in and 200 out, one request costs 0.0139 pence, and 1,200 cost 16.68 pence.

Command
docker exec tools python3 steps/tokens.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Check

Tip

A reply ends with finish_reason length. What happened? Answer: it reached max_tokens and was cut short, so it may be unfinished. Ask for more tokens, or ask a shorter question.

Sampling settings change how a real model chooses

A real model picks each next token from likely ones. temperature near 0 nearly always takes the likeliest; higher takes more chances. seed repeats a choice. The script sends five settings and gets one reply: "the same every time: True". The stand-in ignores them, which a real model would not.

Command
docker exec tools python3 steps/sampling.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Structured output is JSON your program checks

Free text is hard for a program to use. Here the request offers a tool, record_person, with a schema, and forces the model to call it. The answer is tool_calls, with arguments as text: {"name": "Aiko", "age": 17, "city": "Leeds"}. The script parses it, checks every type, then uses the age as a number.

Command
docker exec tools python3 steps/structured.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Streams arrive in pieces, and errors need a plan

With "stream":true the answer comes as 14 events: an empty start, one per token, a finish, the usage, then [DONE]. The script then sends #busy, which the stand-in refuses with 429 two times in three. It waits 1 second, then 2, and the third try works. #broken always fails, so it gives up. A 400 is never retried.

Command
curl -sN http://model:4000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"lab","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"What is the capital of France?"}]}'

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
docker exec tools python3 steps/retry.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A common mistake

Warning

Many people think the model remembers the conversation. It does not: each request stands alone, and the model knows only the messages in it. A chat app sends the whole conversation again every time, which is why long chats use more tokens and cost more.

Exit check

Tip

Answer without looking back. 1. Which role sets the rules for a conversation? 2. What does finish_reason length mean? 3. Why parse and check structured output? 4. Which errors are worth retrying? Answers: system; it hit max_tokens; it is text until your program checks it; busy ones like 429, after a growing pause.

For teachers

Tip

This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is two services of one Python image: a stand-in model that answers in a real model API's format by fixed rules, and a Python with a client and short scripts. No real model is called, so it costs nothing and every output is the same. Key learning points: 1. A request is a list of messages, each with a role. 2. The model remembers only what each request sends. 3. Tokens limit a reply and set its cost. 4. Structured output must be parsed and checked. Common mistake: thinking the model remembers. Equipment: the lab and a browser. Tip: ask students to predict each output before they press Run, then say what a real model would do differently. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.

About this lab and its source: LLM workbench lab: a language model API to practise on, with a stand-in model

Teaching a class? Set these lessons for your students with marking and a lab for every student.