A photographed invoice is not text yet
A shop photographs a paper invoice and wants the total in its database. The photo is only pixels. A reader program has to find the words first.
It returns each piece of text with its place on the page. Your code then puts the pieces in order and pulls out the fields.
By the end of this lesson
Note
I can turn a page of text boxes into lines in reading order and pull out its fields with their confidence.
Key words
| Word | What it means | Example |
|---|---|---|
| Box | a piece of text and where it sits on the page | |
| Reading order | the order a person would read the boxes | |
| Field | a label and the value next to it | Total: 46.50 |
| Confidence | how sure the reader is, from 0 to 100 | |
| Tolerance | how far apart two boxes can be and still share a line |
The cells students fill in are left blank in this preview.
A page is a list of boxes
A page reader does not give you one block of text. It gives a list of boxes. Each box holds some text, its place (x across, y down) and a confidence from 0 to 100.
The list is in no useful order. Two boxes that look like one line may be a few pixels apart.
Four boxes from a page
| Text | x | y | Confidence |
|---|---|---|---|
| Total: | 40 | 200 | 98 |
| 46.50 | 120 | 201 | 61 |
| Name: | 40 | 60 | 97 |
| Asha | 120 | 62 | 95 |
The cells students fill in are left blank in this preview.
Reading order needs a tolerance
To read a page you sort the boxes top to bottom, then group boxes into lines, then sort each line left to right.
Boxes on one line are rarely at exactly the same y. So a box joins a line when its y is within a tolerance of the line's first box. Here, a tolerance of 5 pixels is used.
A common mistake
Warning
Many people sort the boxes by y alone and read them in that order. A value a few pixels higher than its label then comes before it. Group boxes into lines with a tolerance first, then sort each line by x.
A field is a label and its value
In a form, a label such as Total: is followed by its value on the same line. To pull a field, find a box that ends with a colon and take the next box in the line.
Keep the confidence with the value. A low figure means a person should check it, because a misread digit looks exactly like a right one.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const boxes = [' ' { text: \"46.50\", x: 120, y: 201, conf: 61 },' ' { text: \"Total:\", x: 40, y: 200, conf: 98 },' ' { text: \"Invoice\", x: 40, y: 10, conf: 99 },' ' { text: \"Asha\", x: 120, y: 62, conf: 95 },' ' { text: \"Name:\", x: 40, y: 60, conf: 97 },' ' { text: \"12/03/2026\", x: 120, y: 99, conf: 88 },' ' { text: \"Date:\", x: 40, y: 100, conf: 98 },' '];' > page3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function lines(list, tol) {' ' const sorted = list.slice().sort(function (a, b) { return a.y - b.y || a.x - b.x; });' ' const out = [];' ' for (const b of sorted) {' ' const last = out[out.length - 1];' ' if (last && Math.abs(b.y - last[0].y) <= tol) last.push(b);' ' else out.push([b]);' ' }' ' return out.map(function (l) { return l.sort(function (a, b) { return a.x - b.x; }); });' '}' >> page3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function fields(ls) {' ' const found = {};' ' for (const l of ls) {' ' for (let i = 0; i < l.length - 1; i++) {' ' if (l[i].text.endsWith(\":\")) found[l[i].text.slice(0, -1).toLowerCase()] = { value: l[i + 1].text, conf: l[i + 1].conf };' ' }' ' }' ' return found;' '}' >> page3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the page script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const ls = lines(boxes, 5);' ' for (const l of ls) console.log(l.map(function (b) { return b.text; }).join(\" \"));' ' const f = fields(ls);' ' console.log(JSON.stringify(f));' ' const low = Object.keys(f).filter(function (k) { return f[k].conf < 90; });' ' console.log(\"check by hand: \" + low.join(\", \"));' ' const q = \"Write one short question asking a person to confirm a total that was read with low confidence.\";' ' const reply = await ask([{ role: \"user\", content: q }], \"Please check this total against the paper page: is it 46.50?\", 40);' ' console.log(\"reply: \" + reply.text);' '}' 'main();' >> page3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Read the page and pull fields
docker exec -w /home/node/project agent node page3.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The lines it prints
| Line | What it shows |
|---|---|
| Name: Asha | Two boxes joined into one line, label first |
| Date: 12/03/2026 | y 99 and y 100 are inside the tolerance of 5 |
| {"name":{...},"date":{...},"total":{...}} | Each field with its value and confidence |
| check by hand: date, total | The two values below 90 |
The cells students fill in are left blank in this preview.