How to compare two AI versions fairly
An A/B test gives version A to one group of users and version B to another, then compares how often each one works. A shadow test runs the new version on real requests, but nobody sees its answers.
Both can fool you. A gap of nine points can still be noise.
By the end of this lesson
Note
I can compare two versions without being fooled by chance.
Key words
| Word | What it means |
|---|---|
| A/B test | two groups of users, one version each |
| Shadow test | a new version run where nobody sees it |
| Pass rate | the share of requests that worked |
| Gap | one pass rate minus the other, in points |
| Wobble | how far a gap moves by chance |
The cells students fill in are left blank in this preview.
A gap must beat the wobble
Version B passes 138 of 200 and A passes 120 of 200. The gap is 9 points. But two groups of 200 users differ by chance too.
The wobble is how far a gap moves by chance. A rule of thumb: trust a gap only when it is at least twice the wobble.
Worked example: two tests
| Test | Users each | Gap | Wobble | Twice the wobble | Verdict |
|---|---|---|---|---|---|
| A against B | 200 | 9.0 | 4.8 | 9.6 | Not proven |
| A against C | 1,000 | 5.0 | 2.2 | 4.3 | Trustworthy |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think the higher score wins. A higher score from a small test may be chance. Call a winner only when the gap is at least twice the wobble, and otherwise say not proven and collect more data.
A shadow test finds new failures safely
Run the new version on the same real requests, and keep its answers hidden. For each request you have a pair: the old result and the new result.
Equal totals can hide change. Two cases may get worse while two others get better. Count the worse ones, and read each.
Worked example: twelve pairs
| Pair (old, new) | Meaning | Count |
|---|---|---|
| pass, pass | Same | 7 |
| fail, fail | Same | 1 |
| fail, pass | New better | 2 |
| pass, fail | New worse, read these | 2 |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'function wobble(a, an, b, bn) {' ' const p = (a + b) / (an + bn);' ' return Math.sqrt(p * (1 - p) * (1 / an + 1 / bn)) * 100;' '}' 'function gap(a, an, b, bn) { return (b / bn - a / an) * 100; }' 'function verdict(a, an, b, bn) {' ' const ok = Math.abs(gap(a, an, b, bn)) >= 2 * wobble(a, an, b, bn);' ' return ok ? \"trustworthy\" : \"not proven\";' '}' > ab.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const tests = [' ' { name: \"A v B\", a: 120, an: 200, b: 138, bn: 200 },' ' { name: \"A v C\", a: 600, an: 1000, b: 650, bn: 1000 },' '];' 'const pairs = \"pp pp pf pp fp pp ff pp pf pp pp fp\".split(\" \");' 'function count(kind) { return pairs.filter(function (p) { return p === kind; }).length; }' >> ab.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' tests.forEach(function (t) {' ' console.log(t.name + \": gap \" + gap(t.a, t.an, t.b, t.bn).toFixed(1) + \" points, wobble \" + wobble(t.a, t.an, t.b, t.bn).toFixed(1) + \" -> \" + verdict(t.a, t.an, t.b, t.bn));' ' });' ' console.log(\"shadow: same \" + (count(\"pp\") + count(\"ff\")) + \", new better \" + count(\"fp\") + \", new worse \" + count(\"pf\"));' ' const q = \"In one short sentence, why can a 9 point gap still be noise?\";' ' const sample = \"With only 200 users each, chance alone can move a score by that much.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> ab.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the comparison
docker exec -w /home/node/project agent node ab.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| A v B: gap 9.0 points, wobble 4.8 -> not proven | A big gap, a small test |
| A v C: gap 5.0 points, wobble 2.2 -> trustworthy | A smaller gap, a bigger test |
| shadow: same 8, new better 2, new worse 2 | Counted from twelve recorded pairs |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.