Two searches, two blind spots
Hybrid search runs a keyword search and a meaning search, then merges the two ranked lists. It matters because each one misses what the other finds.
A user types "forgot my login". No help page contains "login", so keywords find nothing. Another types "order number 4417". Meaning search has no idea what 4417 is.
By the end of this lesson
Note
I can combine keyword and meaning search into one ranked list and say when each one wins.
Key words
| Word | What it means |
|---|---|
| BM25 | a keyword score that rewards rare words and short documents |
| Term frequency | how often a word appears in one document |
| Inverse document frequency | a weight that is high for words few documents hold |
| Rank | the place of a result in a list, counted from 1 |
| Reciprocal rank fusion | merging lists by adding 1 over k plus the rank |
| Lexical search | search that matches the words themselves |
The cells students fill in are left blank in this preview.
BM25 scores the words themselves
A keyword search counts the query words in each document. BM25 improves on a plain count in three ways. A word counts for more when few documents hold it. Repeating a word helps less each time. A short document beats a long one with the same count.
The result is exact: a word that is not there scores nothing. That is its strength and its blind spot.
Meaning search compares vectors
Meaning search turns each text into a vector, a list of numbers, and compares their direction. Here each help page has three invented numbers: how much it is about accounts, delivery and money.
A query gets numbers of its own, and the closest pages win. "Forgot my login" is mostly about accounts, so account pages come first, even with no word in common. A real system gets its vectors from an embedding model with hundreds of numbers.
Fuse the ranks, never the scores
BM25 scores have no fixed top: one can be 0.5 and another 12. Cosine scores here stay between 0 and 1. Adding them would let one list drown the other.
Reciprocal rank fusion ignores scores and uses ranks. An item gets 1 / (60 + rank) from each list that holds it, and the points add up. An item both lists like beats an item only one list likes.
Worked example: order number 4417
| Page | Keyword rank | Meaning rank | Fused points |
|---|---|---|---|
| d4 track a parcel | 1 | 3 | 0.0323 |
| d3 refund policy | none | 1 | 0.0164 |
| d5 pay by card | none | 2 | 0.0161 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think hybrid search means adding the two scores. The scales differ, so one list wins every time. Add ranks instead, with 1 / (k + rank), and the two lists get an equal say.
Write the hybrid search, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const docs = [' ' { id: \"d1\", text: \"reset your password from the sign in page\", v: [9, 0, 1] },' ' { id: \"d2\", text: \"change the email address on your account\", v: [8, 1, 0] },' ' { id: \"d3\", text: \"refund policy for returned parcels\", v: [1, 5, 8] },' ' { id: \"d4\", text: \"track a parcel with the order number\", v: [0, 9, 1] },' ' { id: \"d5\", text: \"pay by card or bank transfer\", v: [1, 1, 9] },' ' { id: \"d6\", text: \"delete your account and your data\", v: [7, 0, 2] },' '];' 'const K1 = 1.5, B = 0.75;' 'function words(t) { return t.toLowerCase().split(\" \"); }' 'const avg = docs.reduce(function (s, d) { return s + words(d.text).length; }, 0) / docs.length;' > hybrid1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the hybrid search, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function bm25(query, doc) {' ' let score = 0;' ' words(query).forEach(function (w) {' ' const n = docs.filter(function (d) { return words(d.text).includes(w); }).length;' ' if (n === 0) return;' ' const idf = Math.log(1 + (docs.length - n + 0.5) / (n + 0.5));' ' const f = words(doc.text).filter(function (x) { return x === w; }).length;' ' const len = words(doc.text).length;' ' score += idf * f * (K1 + 1) / (f + K1 * (1 - B + B * len / avg));' ' });' ' return score;' '}' >> hybrid1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the hybrid search, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function cosine(a, b) {' ' let dot = 0, sa = 0, sb = 0;' ' for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; sa += a[i] * a[i]; sb += b[i] * b[i]; }' ' return dot / Math.sqrt(sa * sb);' '}' 'function ranked(scores) {' ' return docs.map(function (d, i) { return { id: d.id, s: scores[i] }; })' ' .filter(function (x) { return x.s > 0.01; })' ' .sort(function (x, y) { return y.s - x.s; })' ' .map(function (x) { return x.id; });' '}' 'function fuse(lists) {' ' const total = {};' ' lists.forEach(function (list) {' ' list.forEach(function (id, at) { total[id] = (total[id] || 0) + 1 / (60 + at + 1); });' ' });' ' return Object.keys(total).sort(function (x, y) { return total[y] - total[x]; });' '}' >> hybrid1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the hybrid search, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const queries = [' ' { text: \"forgot my login\", v: [9, 0, 0] },' ' { text: \"order number 4417\", v: [2, 3, 3] },' '];' 'queries.forEach(function (q) {' ' const a = ranked(docs.map(function (d) { return bm25(q.text, d); }));' ' const b = ranked(docs.map(function (d) { return cosine(q.v, d.v); })).slice(0, 3);' ' console.log(q.text);' ' console.log(\" keywords:\", a.join(\" \") || \"nothing\");' ' console.log(\" meaning: \", b.join(\" \"));' ' console.log(\" fused: \", fuse([a, b]).join(\" \"));' '});' >> hybrid1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run and compare the lists
docker exec -w /home/node/project agent node hybrid1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
This is a stand-in
The Run makes no model call. The vectors are three invented numbers and six pages stand for an index. A real system holds millions of pages and gets its vectors from an embedding model.
The shape is the same: two ranked lists, then one fused list.
What the Run prints
| Query | Keywords | Meaning | Fused |
|---|---|---|---|
| forgot my login | nothing | d1 d2 d6 | d1 d2 d6 |
| order number 4417 | d4 | d3 d5 d4 | d4 d3 d5 |
The cells students fill in are left blank in this preview.