A search that tells secrets
Search permissions mean a person only finds what they may read. Freshness means deleted or old text is gone from the index. Both matter because search copies your documents into a new place, and the copy ignores the rules of the original.
A student searches "holiday dates" and gets a payroll page. Or a deleted policy still answers.
By the end of this lesson
Note
I can keep a search inside what each person may see and make a deletion or a new version reach the index.
Key words
| Word | What it means |
|---|---|
| Access filter | a rule that keeps only what a person may read |
| Group | a named set of people who share access |
| Stale | a copy older than the current version |
| Version | a number that rises each time a document changes |
| Tombstone | a record that something was deleted |
| Leak | showing something to someone who may not see it |
The cells students fill in are left blank in this preview.
Filter before you rank, never after
Each piece carries the groups that may read it, a kind of metadata. At query time the search keeps only the pieces the asker's groups allow, and then takes the best k.
Filtering after the top k goes wrong twice. A hidden page uses up a place, so the user gets fewer results than they should. And a count of "3 results, 2 hidden" tells them the pages exist.
Worked example: Ana asks for holiday dates
| Approach | Results for Ana | What goes wrong |
|---|---|---|
| Filter after the top 2 | e1 | A payroll page took a place, then was hidden |
| Filter before the top 2 | e1, e3 | Nothing |
| Filter before, stale dropped | e1 | The old copy e3 is gone |
The cells students fill in are left blank in this preview.
A deletion must reach the index
The index holds its own copy of every document. Deleting the source file leaves that copy, so the old text still answers. Delete every piece from the index, and from any cache of answers, in the same step. Keep a tombstone, a record of the deletion, so an old copy cannot creep back.
A changed document is the same problem. Give each piece a version, and drop older pieces when a document is refreshed.
A common mistake
Warning
Many people think deleting the source file removes the text from search. The index kept a copy, so the text is still found. Delete from the index too, and check by searching for a phrase from the deleted page.
Write the access search, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const entries = [' ' { id: \"e2\", doc: \"payroll\", groups: [\"staff\"], version: 1, text: \"holiday bonus dates for managers\" },' ' { id: \"e4\", doc: \"rota\", groups: [\"staff\"], version: 1, text: \"who works on holiday\" },' ' { id: \"e1\", doc: \"handbook\", groups: [\"staff\", \"students\"], version: 2, text: \"holiday dates for the term\" },' ' { id: \"e3\", doc: \"handbook\", groups: [\"staff\", \"students\"], version: 1, text: \"old holiday dates\" },' '];' 'const current = { handbook: 2, payroll: 1, rota: 1 };' 'const ana = { name: \"ana\", groups: [\"students\"] };' > access1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the access search, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function allowed(e, user) { return e.groups.some(function (g) { return user.groups.includes(g); }); }' 'function score(e, q) { return q.filter(function (w) { return e.text.split(\" \").includes(w); }).length; }' 'function top(list, q, k) {' ' return list.filter(function (e) { return score(e, q) > 0; })' ' .sort(function (a, b) { return score(b, q) - score(a, q); })' ' .slice(0, k);' '}' 'function ids(list) { return list.map(function (e) { return e.id; }).join(\" \") || \"nothing\"; }' >> access1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the access search, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const q = [\"holiday\", \"dates\"];' 'function mine(list) { return list.filter(function (e) { return allowed(e, ana); }); }' 'console.log(\"filter after the top 2:\", ids(mine(top(entries, q, 2))));' 'console.log(\"filter before the top 2:\", ids(top(mine(entries), q, 2)));' 'const fresh = entries.filter(function (e) { return e.version >= current[e.doc]; });' 'console.log(\"and with stale copies dropped:\", ids(top(mine(fresh), q, 2)));' 'const left = entries.filter(function (e) { return e.doc === \"handbook\" ? false : true; });' 'console.log(\"after deleting handbook, ana finds:\", ids(top(mine(left), q, 2)));' >> access1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run and compare the filters
docker exec -w /home/node/project agent node access1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
This is a stand-in
The index is a list of four entries and a person is a name with groups. A real system filters inside its search engine, from the signed-in account, and a deletion is a real call to it.
The Run makes no model call. The rule is the same: filter first, delete everywhere.
What the Run prints
| Line | Result |
|---|---|
| filter after the top 2 | e1 |
| filter before the top 2 | e1 e3 |
| and with stale copies dropped | e1 |
| after deleting handbook, ana finds | nothing |
The cells students fill in are left blank in this preview.