Change a little, keep the rest
A large model has billions of numbers. Training all of them needs a lot of memory and time. An adapter, such as a LoRA, trains a few extra numbers beside the big ones and leaves the big ones alone.
This lesson counts those numbers. No lab here trains a model, so the working is arithmetic you run.
By the end of this lesson
Note
I can say why a small adapter changes a large model cheaply.
Key words
| Word | What it means |
|---|---|
| Parameter | one number a model has learned |
| Frozen | not changed during training |
| Adapter | a small set of new trained numbers |
| Rank | how thin the two adapter tables are |
| LoRA | an adapter made of two thin tables |
The cells students fill in are left blank in this preview.
A layer is a table of numbers
Inside a model, a layer is mostly a table of numbers. A table with 6 rows and 6 columns holds 6 times 6, which is 36 parameters. A table 100 by 100 holds 10,000.
Full training changes every number in every table. That is why it is costly.
Two thin tables make a full-size change
Multiply a table that is 6 by 1 by one that is 1 by 6 and you get a 6 by 6 table. It has the shape of the frozen table, so it can be added to it.
The adapter holds 6 plus 6, which is 12 numbers, instead of 36. The rank is the thin side. A bigger rank holds more and costs more.
Worked example: one 100 by 100 table
| Rank | Adapter numbers | Share of the 10,000 |
|---|---|---|
| 1 | 1 x (100 + 100) = 200 | 2% |
| 4 | 4 x 200 = 800 | 8% |
| 8 | 8 x 200 = 1,600 | 16% |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think an adapter is a smaller copy of the model. It is not. The big model stays, unchanged, and the adapter is added to its output. Take the adapter away and you have the original model back.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the adapter counter, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '// A frozen 6 by 6 table W, and two thin ones: B is 6 by 1 and A is 1 by 6.' 'const rows = 6, cols = 6, rank = 1;' 'const B = [1, 2, 0, 1, 0, 3];' 'const A = [2, 0, 1, 0, 1, 1];' > adapt4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the adapter counter, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function delta(i, j) {' ' let sum = 0;' ' for (let k = 0; k < rank; k++) sum = sum + B[i * rank + k] * A[k * cols + j];' ' return sum;' '}' 'const full = rows * cols;' 'const adapter = rank * (rows + cols);' >> adapt4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the adapter counter, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"frozen table: \" + rows + \" x \" + cols + \" = \" + full + \" numbers\");' ' console.log(\"adapter: \" + rows + \" x \" + rank + \" plus \" + rank + \" x \" + cols + \" = \" + adapter + \" numbers\");' ' const first = [];' ' for (let j = 0; j < cols; j++) first.push(delta(0, j));' ' console.log(\"change to row 0: \" + first.join(\" \"));' ' console.log(\"trained share: \" + Math.floor((adapter * 100) / full) + \"%\");' ' const q = \"In one sentence, why is training a small adapter cheaper than training a whole model?\";' ' const sample = \"Only the small adapter tables change, so far fewer numbers are stored and updated.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 40);' ' console.log(\"model: \" + reply.text);' '}' 'main();' >> adapt4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the adapter counter
docker exec -w /home/node/project agent node adapt4.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the counter printed
It prints the size of the frozen table, the size of the adapter, one row of the change and the trained share. The numbers in A and B are invented. A real adapter learns them.
The last line is one short model call, or a Recorded sample.