A photo is only numbers to a computer
A computer cannot look at a photo. It stores a grid of numbers, one for each dot. A model that takes pictures reads those numbers.
This matters because the bigger the grid, the more it costs to send. This lesson shows you the grid and the price of it.
By the end of this lesson
Note
I can say how an image becomes numbers a model reads, and what size and tiles do to its cost.
Key words
| Word | What it means |
|---|---|
| Pixel | one dot of a picture, stored as a number |
| Channel | one colour part of a pixel: red, green or blue |
| Resize | change the width and height of a picture |
| Tile | a small square piece of a picture |
| Multimodal | taking more than one kind of input, such as text and pictures |
The cells students fill in are left blank in this preview.
A picture is a grid of numbers
A picture is split into rows of tiny dots called pixels. Each pixel is stored as a number from 0 (black) to 255 (white) for a grey picture.
A colour picture keeps three numbers for every pixel, one for each channel: red, green and blue. So a colour picture holds three times the numbers of a grey one the same size.
A grey picture and a colour pixel
| Row | Pixel 1 | Pixel 2 | Pixel 3 |
|---|---|---|---|
| 1 | 0 | 128 | 255 |
| 2 | 64 | 128 | 192 |
| 3 | 255 | 128 | 0 |
| A colour pixel | red 200 | green 30 | blue 30 |
The cells students fill in are left blank in this preview.
Big pictures are made smaller first
A model only accepts pictures up to a size. A bigger one is resized before it is read, or it is refused.
The simplest way to shrink a grid is to average each block of pixels into one number. A 2 by 2 block of four pixels becomes one pixel, and the picture is half as wide and half as tall.
A common mistake
Warning
Many people think a smaller file always means a cheaper picture. Cost follows the width and height in pixels, not the file size on disk. A tiny file can still be a huge grid, so always count pixels.
Tiles set the cost
After resizing, the picture is cut into square tiles. A tile at the edge may be only partly filled, but it still counts as a whole tile.
In this lab, a start fee of 20 and 10 for each tile of 16 by 16 pixels is the rule. It is invented for the lesson. A real service has its own rule, so always read its price page.
Counting tiles of 16 by 16
| Picture | Tiles across | Tiles down | Tiles | Cost |
|---|---|---|---|---|
| 16 by 16 | 1 | 1 | 1 | 30 |
| 17 by 16 | 2 | 1 | 2 | 40 |
| 40 by 30 | 3 | 2 | 6 | 80 |
| 64 by 64 | 4 | 4 | 16 | 180 |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the picture script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'let seed = 7;' 'function next() { seed = (seed * 37 + 11) % 256; return seed; }' 'const image = [];' 'for (let y = 0; y < 6; y++) {' ' const row = [];' ' for (let x = 0; x < 6; x++) row.push(next());' ' image.push(row);' '}' > see1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the picture script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function shrink(img, f) {' ' const out = [];' ' for (let y = 0; y < img.length; y += f) {' ' const row = [];' ' for (let x = 0; x < img[0].length; x += f) {' ' let sum = 0;' ' for (let j = 0; j < f; j++) for (let i = 0; i < f; i++) sum += img[y + j][x + i];' ' row.push(Math.floor(sum / (f * f)));' ' }' ' out.push(row);' ' }' ' return out;' '}' 'function tiles(w, h, t) { return Math.ceil(w / t) * Math.ceil(h / t); }' >> see1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the picture script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const small = shrink(image, 2);' ' console.log(\"big \" + image[0].length + \"x\" + image.length + \" row 1: \" + image[0].join(\" \"));' ' console.log(\"small \" + small[0].length + \"x\" + small.length + \" row 1: \" + small[0].join(\" \"));' ' console.log(\"tiles of 2: big \" + tiles(6, 6, 2) + \", small \" + tiles(3, 3, 2));' ' const q = \"In one short sentence, why does a smaller picture cost less to send to a model?\";' ' const reply = await ask([{ role: \"user\", content: q }], \"A smaller picture has fewer tiles, and the price follows the tiles.\", 40);' ' console.log(\"reply: \" + reply.text);' '}' 'main();' >> see1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Shrink and tile a picture
docker exec -w /home/node/project agent node see1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The lines it prints
| Line | What it shows |
|---|---|
| big 6x6 row 1: 14 17 128 139 34 245 | The first row of the made-up picture |
| small 3x3 row 1: 88 166 132 | Each 2 by 2 block averaged into one pixel |
| tiles of 2: big 9, small 4 | Fewer tiles after shrinking, so a lower price |
| reply: ... | A sentence from the model, or a Recorded sample line first |
The cells students fill in are left blank in this preview.