A row with no header
Chunk quality means a piece of a document can still be found and understood on its own. It matters because search returns pieces, and a piece that lost its heading or its table header says nothing.
A fee table is cut into rows. The row "large, 5 pounds" is returned for "how much is a large parcel". It does not say what the 5 pounds is for.
By the end of this lesson
Note
I can split a structured document so that each piece keeps its heading, its table header and its source.
Key words
| Word | What it means |
|---|---|
| Chunk | one piece of a document that is indexed |
| Heading path | the titles above a piece, from the top down |
| Metadata | facts kept beside a piece, such as its source |
| Table header | the first row, which names each column |
| Breadcrumb | a heading path written as one line |
| Orphan | a piece that means nothing without its surroundings |
The cells students fill in are left blank in this preview.
Keep the heading path with every piece
A simple splitter cuts a document by size or at its headings. The next step is to keep the headings. A piece under "Returns > Fees" should carry that path, written as a breadcrumb: Shop handbook > Returns > Fees.
Index the path with the text. A search for "returns fees" then finds the piece even if the body never says "returns".
A table row needs its header
A table is a header and rows. Cut it row by row and each row loses the header. So turn each row into words that carry it: "item: large, fee: 5 pounds".
A list is the same. Keep the sentence that introduces it with each item, or the item floats free.
Worked example: the same row, before and after
| Version | What is indexed |
|---|---|
| Cut loose | | large | 5 pounds | |
| With its header | item: large, fee: 5 pounds |
| With the path too | Shop handbook > Returns > Fees: item: large, fee: 5 pounds |
The cells students fill in are left blank in this preview.
Metadata and a size check
Keep metadata beside each piece: its source file, its heading and later its owner and its date. Metadata lets an answer cite where it came from. It also lets search filter pieces by who may see them.
Then check size. A section far longer than the rest is two ideas and should be split again. Report the headings that are too long.
A common mistake
Warning
Many people think cutting every 500 characters is good enough for any document. It cuts through tables and headings, and leaves orphans. Cut at the structure first, and use size only to split what is still too long.
Write the splitter, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const doc = [' ' \"# Shop handbook\",' ' \"## Opening hours\",' ' \"The shop opens at nine on weekdays.\",' ' \"## Returns\",' ' \"### Time limit\",' ' \"Return a parcel within thirty days.\",' ' \"### Fees\",' ' \"| item | fee |\",' ' \"| small | 2 pounds |\",' ' \"| large | 5 pounds |\",' '];' > chunk1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the splitter, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function cells(line) {' ' return line.split(\"|\").map(function (c) { return c.trim(); }).filter(function (c) { return c.length > 0; });' '}' >> chunk1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the splitter, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function chunks(lines, source) {' ' const path = [];' ' const out = [];' ' let head = null;' ' lines.forEach(function (line) {' ' const m = line.match(/^(#+) (.*)/);' ' if (m) {' ' path.length = m[1].length - 1;' ' path.push(m[2]);' ' out.push({ source: source, heading: path.join(\" > \"), rows: [] });' ' head = null;' ' return;' ' }' >> chunk1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the splitter, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' const now = out[out.length - 1];' ' if (line.startsWith(\"|\") === false) { now.rows.push(line); return; }' ' const c = cells(line);' ' if (head === null) { head = c; return; }' ' now.rows.push(head.map(function (h, i) { return h + \": \" + c[i]; }).join(\", \"));' ' });' ' return out;' '}' >> chunk1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the splitter, part 5
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'console.log(\"A table row cut loose:\", doc[9]);' 'chunks(doc, \"handbook.md\").filter(function (c) { return c.rows.length > 0; }).forEach(function (c) {' ' console.log(c.source + \" | \" + c.heading + \" | \" + c.rows.join(\" / \"));' '});' >> chunk1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run and read the chunks
docker exec -w /home/node/project agent node chunk1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
This is a stand-in
The Run reads ten lines of invented Markdown. A real document may be a PDF, a web page or a spreadsheet, and needs a parser for each. The idea is the same: find the structure, then keep it.
The Run makes no model call.
What the Run prints
| Chunk | Text kept |
|---|---|
| Shop handbook > Opening hours | The shop opens at nine on weekdays. |
| Shop handbook > Returns > Time limit | Return a parcel within thirty days. |
| Shop handbook > Returns > Fees | item: small, fee: 2 pounds / item: large, fee: 5 pounds |
The cells students fill in are left blank in this preview.