AI glossary
Quantisation
Quantisation stores a model's weights with fewer bits, such as 4 instead of 16, so it needs less memory and runs faster with a small loss of quality.
A model of 7 billion parameters at 16 bits needs about 14 GB just for its weights. At 4 bits it needs about 4 GB, which fits on an ordinary laptop or a smaller graphics card.
The loss is usually small at 8 bits and noticeable for some tasks at 4 bits or below. Measure it on your own task rather than trusting a general benchmark.
Weights are not the only memory. The context, held as a key-value cache, grows with every token, so a long prompt can still run out of memory on a quantised model.
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.