Better by 13 points, or by luck?
Statistical significance asks whether a difference between two agents is bigger than chance would give. It matters because an agent differs on every run, so one pass rate for each version can point the wrong way.
The new version of the shop agent scores 13 points higher. Do you ship it?
By the end of this lesson
Note
I can say whether one agent is better than another when each run differs.
Key words
| Word | What it means |
|---|---|
| Pass at k | a case passed in at least one of k runs |
| Pass every time | a case passed in all k runs |
| Interval | the range a rate could really lie in |
| Sign test | a test on how many cases got better or worse |
| P value | the chance of a gap this big by luck alone |
The cells students fill in are left blank in this preview.
Run every case several times
Run each case k times for each version, here five. Two numbers describe a case. Pass at k says it passed at least once, which shows the agent can do it. Pass every time says it passed in all k runs, which shows it can be trusted.
A support agent that must pass every time is a different bar from one that must pass sometimes.
Worked example: eight cases, five runs each
| Version | Rate | Pass at 5 | Pass every time |
|---|---|---|---|
| Old | 60 percent | 7 of 8 | 2 of 8 |
| New | 73 percent | 8 of 8 | 3 of 8 |
The cells students fill in are left blank in this preview.
Ask how much of the gap is chance
A rate from 40 runs is an estimate. Draw an interval around it: about two standard errors either side. A standard error is the usual size of chance in that rate. If the old and new intervals overlap a lot, the gap may be luck.
More runs and more cases narrow the interval. That is the only honest way to make a small gap clear.
Count the cases that moved
The sign test ignores how big each change is. It counts the cases that got better and the cases that got worse. Ties do not count.
The p value is the chance of a split this lopsided if the versions were really equal. Choose the limit first. Here it is 0.05, and a p value under it says better or worse.
Write the recorded runs
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Five recorded runs of each case: 1 passed, 0 failed.' 'const oldVersion = [\"11111\", \"11101\", \"10100\", \"11011\", \"00000\", \"01101\", \"11111\", \"00010\"];' 'const newVersion = [\"11111\", \"11111\", \"10101\", \"11110\", \"00100\", \"11011\", \"11111\", \"01010\"];' 'module.exports = { oldVersion, newVersion };' > runs.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { oldVersion, newVersion } = require(\"./runs\");' 'function passes(text) {' ' return text.split(\"\").filter(function (c) { return c === \"1\"; }).length;' '}' > stats.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function summary(name, version) {' ' const counts = version.map(passes);' ' const total = counts.reduce(function (a, b) { return a + b; }, 0);' ' const runs = version.length * version[0].length;' ' const p = total / runs;' ' const two = 2 * Math.sqrt(p * (1 - p) / runs);' ' const atLeast = counts.filter(function (n) { return n > 0; }).length;' ' const every = counts.filter(function (n) { return n === version[0].length; }).length;' ' console.log(name + \": rate \" + Math.round(p * 100) + \" percent, interval \" + Math.round((p - two) * 100) + \" to \" + Math.round((p + two) * 100));' ' console.log(name + \": pass at 5 \" + atLeast + \" of \" + version.length + \", pass every time \" + every + \" of \" + version.length);' '}' 'summary(\"Old\", oldVersion);' 'summary(\"New\", newVersion);' >> stats.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function signP(better, worse) {' ' const n = better + worse;' ' if (n === 0) return 1;' ' const low = Math.min(better, worse);' ' let total = 0;' ' let ways = 1;' ' for (let i = 0; i <= low; i = i + 1) {' ' total = total + ways;' ' ways = ways * (n - i) / (i + 1);' ' }' ' return Math.min(1, 2 * total / Math.pow(2, n));' '}' >> stats.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'let better = 0;' 'let worse = 0;' 'oldVersion.forEach(function (text, at) {' ' const a = passes(text);' ' const b = passes(newVersion[at]);' ' if (b > a) better = better + 1;' ' if (b < a) worse = worse + 1;' '});' 'const p = signP(better, worse);' 'console.log(\"Cases better: \" + better + \", worse: \" + worse + \", tied: \" + (oldVersion.length - better - worse));' 'console.log(\"Sign test p: \" + p.toFixed(4));' 'console.log(\"Verdict: \" + (p < 0.05 ? (better > worse ? \"better\" : \"worse\") : \"unclear\"));' >> stats.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Compare the two versions
docker exec -w /home/node/project agent node stats.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the comparison prints
| Line | What it shows |
|---|---|
| Old rate | 60 percent, interval 45 to 75 |
| New rate | 73 percent, interval 58 to 87 |
| Cases | better 5, worse 0, tied 3 |
| Sign test p | 0.0625 |
| Verdict | unclear |
The cells students fill in are left blank in this preview.
Try it
Tip
In runs.js change the fourth case of newVersion from "11110" to "11111", then compare again. Six cases are now better and none worse, so p drops under 0.05.
A common mistake
Warning
Many people run the comparison again and again until the p value drops under 0.05, and then stop. Each run is another chance to be lucky. Fix the number of cases and runs first, run once and report whatever comes out.