Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Int8 quantisation by hand

After this lesson: I can quantise a row of weights to int8 and measure the error.

60 min lesson2 commandsLLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

A quarter of the bytes

Int8 quantisation stores each weight of a language model as a whole number from -127 to 127 in one byte. A scale turns it back into a real number.

Decode reads every weight on every pass, so a quarter of the bytes means a quarter of the reading. The question is what it costs in accuracy.

By the end of this lesson

Note

I can quantise a row of weights to int8 and measure the error.

Key words

WordWhat it means
Int8a whole number stored in one byte
Scalethe number an int8 value is multiplied by to get it back
Quantiseturn real numbers into int8 values with a scale
Dequantisemultiply int8 values by the scale again
Per-row scaleone scale for each row of a matrix
Outliera value far bigger than the rest

The cells students fill in are left blank in this preview.

A scale maps each row onto int8

Find the biggest absolute value in the row. Divide it by 127 to get the scale. Divide every number by the scale and round it: that is its int8 value.

To get a number back, multiply its int8 value by the scale. The biggest value comes back exactly. Every other value comes back a little off, by at most half a scale.

Worked example: one row of four weights

WeightDivided by scale 0.00945Int8Back againError
0.3031.75320.302360.00236
-1.20-127-127-1.20
0.077.4170.066140.00386
0.5052.92530.500790.00079

The cells students fill in are left blank in this preview.

Try it: quantise a row

Code: python
# One row of weights, quantised to int8 with its own scale.
row = [0.30, -1.20, 0.07, 0.50]
biggest = 0
for x in row:
    biggest = max(biggest, abs(x))
scale = biggest / 127
ints = []
for x in row:
    ints.append(round(x / scale))
print("scale:", round(scale, 5))
print("int8 values:", ints)
back = []
for n in ints:
    back.append(round(n * scale, 5))
print("back again:", back)
# Change 0.07 to 0.004. What happens to it?

Quantise the whole model and compare

The lab quantises each of the model's five weight matrices, one scale per row. It prints the worst error in each matrix and the bytes before and after, scales included.

Then it asks the real question: does the model still choose the same next character? It checks every place in the first 64 characters of its text, then writes a reply both ways.

Quantise the model to int8

Command
docker exec jupyter python3 steps/quantise.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

What the Run printed

MeasureResult
Bytes as float3212,108
Bytes as int8, scales included3,543
Worst error in one weight0.00221 to 0.03771
Same next character64 of 64 places
Reply to "the cat ", both wayssat on the mat. the dog

The cells students fill in are left blank in this preview.

One big value spoils a shared scale

With one scale for a whole matrix, the biggest value in the matrix sets the step between levels. If one weight is far bigger than the rest, the step gets wide and small weights lose their detail.

A scale for each row keeps the damage to that one row. The lab sets one weight to 40 and measures the average error both ways.

One scale against a scale per row

Command
docker exec jupyter python3 steps/outlier.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: average error in the last matrix

MatrixOne scaleA scale per row
As trained0.01940.00617
One value set to 400.07820.00702

The cells students fill in are left blank in this preview.

A common mistake

Warning

Many people think int8 makes a model faster because whole-number sums are quicker. On decode the gain comes mostly from reading a quarter of the bytes. A machine that turns int8 back into float32 for every sum still reads less, so it still speeds up.

About this lab and its source: LLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.