Forty policies and the same questions
A document assistant answers questions from a company's own files, and says which file it used. It saves staff searching 40 policy documents for the same answers every day.
In this capstone you plan such a feature and decide how to check it. You work over a recorded sample, so the numbers are fixed and you can compare.
By the end of this lesson
Note
I can plan and check a feature that answers from documents.
Key words
| Word | What it means | Example |
|---|---|---|
| Spec | a short written plan of what a feature must do | |
| Retrieval | finding the passages that hold the answer | |
| Source | the document an answer came from | leave.pdf |
| Refusal | saying the documents do not answer it | |
| Acceptance number | the score a feature must reach before release |
The cells students fill in are left blank in this preview.
Where to go deeper
Note
This capstone plans and checks; it does not teach retrieval. The AI lessons on the Resources page do. Retrieval-augmented generation. Evals and testing for model-backed features. Retrieval systems.
A spec says what good looks like
Before any code, write a spec of half a page. Say who uses the feature and which documents it may read. Say what every answer must show, such as its source.
Say what it must never do. For a document assistant, the big one is guessing. When no document answers, it gives a refusal.
Worked example: the spec on one page
| Part of the spec | What we wrote |
|---|---|
| Who uses it | Office staff, signed in |
| What it may read | The 40 policy documents, nothing else |
| Every answer must | Name the document it came from |
| It must never | Guess when no document answers |
| Acceptance numbers | 18 of 20 right, and every no-answer question refused |
The cells students fill in are left blank in this preview.
An eval set is fixed questions
The eval set turns the spec into cases. Each case is a question and the document that should answer it. Some cases have no document on purpose, to test the refusal.
Keep the set fixed, so two runs can be compared. The program checks a recorded sample of five cases.
Try it: check a recorded sample
# A recorded sample: what the assistant answered in testing.
# "source" is the document the answer should come from.
# An empty source means no document answers it: refuse.
cases = [
{"question": "Days of annual leave?", "source": "leave.pdf", "got": "leave.pdf"},
{"question": "How to claim a taxi?", "source": "expenses.pdf", "got": "expenses.pdf"},
{"question": "Who fixes my laptop?", "source": "it-help.pdf", "got": "leave.pdf"},
{"question": "What is the CEO's salary?", "source": "", "got": ""},
{"question": "Can I bring my dog?", "source": "", "got": "office.pdf"},
]
right = 0
for case in cases:
if case["got"] == case["source"]:
right = right + 1
else:
print("Check:", case["question"])
print("Right:", right, "of", len(cases))
# Change the laptop answer to "it-help.pdf" and run again.
Acceptance numbers decide release
An acceptance number is written in the spec before testing starts. Here it is two numbers: at least 18 of 20 right, and no guesses at all.
Two numbers matter because one can hide the other. A feature can score 19 of 20 and still invent an answer about pay. That one guess blocks release.
A common mistake
Warning
Many people think a few questions tried by hand show the feature works. A handful of tries hides the rare failures. Use a fixed eval set of at least 20 cases, include questions with no answer, and run it after every change.