Will this model fit on my machine?
The memory a model needs is the space its numbers take, plus the space for the prompt it is reading. You can work it out with a short sum before you download anything.
That matters because a model that does not fit simply will not start. A wrong guess wastes an afternoon and sometimes the price of a bigger machine.
By the end of this lesson
Note
I can work out how much memory a model needs.
Key words
| Word | What it means | Example |
|---|---|---|
| Parameter | one number the model learnt | |
| Weights | all the parameters together | |
| Precision | how many bits store each number | 16 bits |
| Cache | memory that holds the prompt read so far | |
| Gigabyte (GB) | a thousand million bytes |
The cells students fill in are left blank in this preview.
Weights are parameters times bytes
A model is mostly a very long list of numbers. Each number is a parameter. Every parameter is stored in some number of bits, and 8 bits make one byte.
So the weights take parameters times bytes each. A model of 8 billion parameters at 16 bits (2 bytes) needs 16 gigabytes for its weights alone.
Model A, weights only
| Precision | Bytes each | Weights of 8 billion |
|---|---|---|
| 32 bits | 4 | 32 GB |
| 16 bits | 2 | 16 GB |
| 8 bits | 1 | 8 GB |
| 4 bits | 0.5 | 4 GB |
The cells students fill in are left blank in this preview.
A long prompt adds a cache
While a model reads, it keeps notes on every token so far. These notes are the cache. One token needs 2 times the layers times the width times the bytes.
For Model A that is 2 x 32 x 1024 x 2 = 131,072 bytes a token. At 8,000 tokens the cache is about 1 GB. The cache grows with the prompt and the weights do not.
Model A, cache by prompt length
| Tokens in the prompt | Cache |
|---|---|
| 1,000 | 0.13 GB |
| 8,000 | 1.05 GB |
| 32,000 | 4.19 GB |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a model that is 16 GB to download needs 16 GB to run. The weights are only part. The cache grows with the prompt, and a machine needs some memory spare. Add the cache, then keep a fifth free.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the memory sizer, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const model = { billions: 8, layers: 32, kvWidth: 1024 };' 'const BITS = [32, 16, 8, 4];' 'function weightsGB(bits) {' ' return model.billions * bits / 8;' '}' > mem.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the memory sizer, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function cacheGB(tokens) {' ' const perToken = 2 * model.layers * model.kvWidth * 2;' ' return perToken * tokens / 1000000000;' '}' >> mem.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the memory sizer, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"Model A: 8 billion parameters, cache kept at 16 bits\");' ' for (const bits of BITS) {' ' const total = weightsGB(bits) + cacheGB(8000);' ' console.log(bits + \" bits: weights \" + weightsGB(bits) + \" GB, total \" + total.toFixed(2) + \" GB\");' ' }' ' console.log(\"Cache at 1000 tokens: \" + cacheGB(1000).toFixed(2) + \" GB\");' ' console.log(\"Cache at 32000 tokens: \" + cacheGB(32000).toFixed(2) + \" GB\");' >> mem.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the memory sizer, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' const q = \"In one short sentence, why does a longer prompt need more memory?\";' ' const sample = \"Each token of the prompt adds to the cache the model keeps, so a longer prompt means a bigger cache.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> mem.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the memory sizer
docker exec -w /home/node/project agent node mem.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Nothing is downloaded
This lab has no GPU and holds no model files. Model A is invented, and every size is a sum. The one short call to the lab's model only words a sentence.
If no model answers, the Run prints Recorded sample first, and the sums are the same.
What the Run prints
| Line | What it shows |
|---|---|
| 32 bits: weights 32 GB, total 33.05 GB | Full size does not fit a small machine |
| 4 bits: weights 4 GB, total 5.05 GB | Fewer bits, much less memory |
| Cache at 32000 tokens: 4.19 GB | A long prompt is a real cost |
| model said: ... | A sentence. The words differ each run |
The cells students fill in are left blank in this preview.