A quarter of the bytes
Int8 quantisation stores each weight of a language model as a whole number from -127 to 127 in one byte. A scale turns it back into a real number.
Decode reads every weight on every pass, so a quarter of the bytes means a quarter of the reading. The question is what it costs in accuracy.
By the end of this lesson
Note
I can quantise a row of weights to int8 and measure the error.
Key words
| Word | What it means |
|---|---|
| Int8 | a whole number stored in one byte |
| Scale | the number an int8 value is multiplied by to get it back |
| Quantise | turn real numbers into int8 values with a scale |
| Dequantise | multiply int8 values by the scale again |
| Per-row scale | one scale for each row of a matrix |
| Outlier | a value far bigger than the rest |
The cells students fill in are left blank in this preview.
A scale maps each row onto int8
Find the biggest absolute value in the row. Divide it by 127 to get the scale. Divide every number by the scale and round it: that is its int8 value.
To get a number back, multiply its int8 value by the scale. The biggest value comes back exactly. Every other value comes back a little off, by at most half a scale.
Worked example: one row of four weights
| Weight | Divided by scale 0.00945 | Int8 | Back again | Error |
|---|---|---|---|---|
| 0.30 | 31.75 | 32 | 0.30236 | 0.00236 |
| -1.20 | -127 | -127 | -1.2 | 0 |
| 0.07 | 7.41 | 7 | 0.06614 | 0.00386 |
| 0.50 | 52.92 | 53 | 0.50079 | 0.00079 |
The cells students fill in are left blank in this preview.
Try it: quantise a row
# One row of weights, quantised to int8 with its own scale.
row = [0.30, -1.20, 0.07, 0.50]
biggest = 0
for x in row:
biggest = max(biggest, abs(x))
scale = biggest / 127
ints = []
for x in row:
ints.append(round(x / scale))
print("scale:", round(scale, 5))
print("int8 values:", ints)
back = []
for n in ints:
back.append(round(n * scale, 5))
print("back again:", back)
# Change 0.07 to 0.004. What happens to it?
Quantise the whole model and compare
The lab quantises each of the model's five weight matrices, one scale per row. It prints the worst error in each matrix and the bytes before and after, scales included.
Then it asks the real question: does the model still choose the same next character? It checks every place in the first 64 characters of its text, then writes a reply both ways.
Quantise the model to int8
docker exec jupyter python3 steps/quantise.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
What the Run printed
| Measure | Result |
|---|---|
| Bytes as float32 | 12,108 |
| Bytes as int8, scales included | 3,543 |
| Worst error in one weight | 0.00221 to 0.03771 |
| Same next character | 64 of 64 places |
| Reply to "the cat ", both ways | sat on the mat. the dog |
The cells students fill in are left blank in this preview.
One big value spoils a shared scale
With one scale for a whole matrix, the biggest value in the matrix sets the step between levels. If one weight is far bigger than the rest, the step gets wide and small weights lose their detail.
A scale for each row keeps the damage to that one row. The lab sets one weight to 40 and measures the average error both ways.
One scale against a scale per row
docker exec jupyter python3 steps/outlier.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: average error in the last matrix
| Matrix | One scale | A scale per row |
|---|---|---|
| As trained | 0.0194 | 0.00617 |
| One value set to 40 | 0.0782 | 0.00702 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think int8 makes a model faster because whole-number sums are quicker. On decode the gain comes mostly from reading a quarter of the bytes. A machine that turns int8 back into float32 for every sum still reads less, so it still speeds up.