Smaller numbers, smaller model
Quantising a model means storing each of its numbers with fewer bits, so the model takes less memory. The price is that every number is rounded.
That matters because rounding is how a big model is squeezed onto a small machine. You need to know what the squeeze costs before you trust the result.
By the end of this lesson
Note
I can say what is lost when numbers get smaller.
Key words
| Word | What it means |
|---|---|
| Quantise | to round numbers onto a few fixed levels |
| Level | one of the values a number can be rounded to |
| Scale | the biggest absolute value in the table |
| Error | the distance between a number and its rounded copy |
| Outlier | a value far bigger than the rest |
The cells students fill in are left blank in this preview.
Fewer bits means fewer levels
With 8 bits a number can sit on 255 levels. With 4 bits there are only 15, and with 2 bits only 3. Each number is moved to the nearest level.
To pick the levels, divide each value by the scale (the biggest absolute value), multiply by the top level, and round. To get a number back, reverse the steps.
Eight numbers at four precisions
| Bits | Levels | Whole numbers kept | Worst error |
|---|---|---|---|
| 8 | 255 | 127 -59 10 71 -124 44 -7 90 | 0.0031 |
| 4 | 15 | 7 -3 1 4 -7 2 0 5 | 0.0586 |
| 3 | 7 | 3 -1 0 2 -3 1 0 2 | 0.12 |
| 2 | 3 | 1 0 0 1 -1 0 0 1 | 0.42 |
The cells students fill in are left blank in this preview.
Where rounding hurts most
Small numbers hurt first. At 4 bits the value -0.05 became 0, so it was lost. At 2 bits four of the eight numbers became 0.
One outlier hurts everyone. If one value is 9.0, the scale is 9.0 and every level is far apart, so all the small numbers round to 0.
A common mistake
Warning
Many people think fewer bits just makes everything a little blurry. It is not even. Small numbers can vanish and one outlier can spoil the whole table. Always measure the error before you trust a quantised table.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the quantiser, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const values = [0.9, -0.42, 0.07, 0.5, -0.88, 0.31, -0.05, 0.64];' 'function quantise(list, bits) {' ' const top = Math.pow(2, bits - 1) - 1;' ' const scale = Math.max(...list.map(Math.abs));' ' return list.map(function (v) { return Math.floor(v / scale * top + 0.5); });' '}' > quant.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the quantiser, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function restore(q, list, bits) {' ' const top = Math.pow(2, bits - 1) - 1;' ' const scale = Math.max(...list.map(Math.abs));' ' return q.map(function (n) { return n * scale / top; });' '}' >> quant.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the quantiser, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function worst(list, bits) {' ' const back = restore(quantise(list, bits), list, bits);' ' let w = 0;' ' list.forEach(function (v, i) { w = Math.max(w, Math.abs(v - back[i])); });' ' return w;' '}' >> quant.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the quantiser, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' for (const bits of [8, 4, 3, 2]) {' ' console.log(bits + \" bits: \" + quantise(values, bits).join(\" \") + \" | worst error \" + worst(values, bits).toFixed(4));' ' }' ' console.log(\"with one 9.0 added, 4 bits, worst error \" + worst(values.concat([9.0]), 4).toFixed(4));' ' const q = \"In one short sentence, what is lost when numbers are stored with fewer bits?\";' ' const sample = \"Each number is rounded to a nearby step, so small differences between numbers are lost.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> quant.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the quantiser
docker exec -w /home/node/project agent node quant.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| 8 bits: ... worst error 0.0031 | Almost nothing is lost |
| 4 bits: 7 -3 1 4 -7 2 0 5 ... | The 0.05 became 0 |
| 2 bits: ... worst error 0.4200 | Most small values are gone |
| with one 9.0 added ... | One outlier makes the error large |
| model said: ... | A sentence. The words differ each run |
The cells students fill in are left blank in this preview.