A shop helper that answers questions
A bike shop builds a helper. A customer types a question. The app finds the best paragraph in the shop's answers and hands it to a language model to word the reply.
If the app finds the wrong paragraph, the model words a wrong answer very politely. This lesson tests the finding. It is plain code, so you can score it without calling any model.
By the end of this lesson
Note
I can test an app that answers questions with a small set of questions, each with the fact the answer must contain.
Key words
| Word | What it means | Example |
|---|---|---|
| Retrieval | finding the text that should help answer a question | |
| Expected fact | a short piece of text that a good answer must contain | twelve pounds |
| Hit rate | the share of questions where the right text was found | |
| Stop word | a common word, such as "the", that tells you little | |
| Wrong answer | text was found, but it does not hold the fact | |
| No answer | nothing matched, so the app says it does not know |
The cells students fill in are left blank in this preview.
Two steps, one easy to test
Many apps answer in two steps. First retrieval: pick the paragraph that matches the question. Then a language model words the reply from that paragraph.
A model's reply changes from run to run, so it is hard to score. Retrieval gives the same result every time, so it is easy to score. If retrieval is wrong, nothing after it can be right.
The simplest retrieval: count shared words
Here the matching is as plain as it can be. Break the question into words. For each paragraph, count how many of those words are also in the paragraph. The paragraph with the highest count wins.
It matches whole words only, so "open" does not match "opens". Run the code and read the counts.
Count the shared words for one question
def words(text):
for mark in ".,?":
text = text.replace(mark, "")
return text.lower().split()
docs = [
"The shop opens at nine and closes at six from Monday to Saturday.",
"A bike costs twelve pounds to hire for a day and five pounds for an hour.",
"You can return a bike to any of the three stations in the city.",
"If a bike breaks, call the number on the frame and we will fix it.",
"Helmets are free with every hire but you must ask at the desk.",
]
question = "What is the cost of a day?"
for text in docs:
shared = []
for word in words(question):
if word in words(text):
shared.append(word)
print(len(shared), shared, "<-", text[:28])
# Change the question to "Do I get a helmet?" and run again.
A test set: questions and expected facts
To score the app you need a small test set. It holds questions a customer might ask. Each has the expected fact that the right paragraph holds.
A question counts as a hit if the paragraph found contains its fact. It is wrong if a paragraph was found without it. It is no answer if nothing matched at all.
Write the shop's five answers into the lab
docker exec jupyter sh -c "printf '%s\n' 'd1|The shop opens at nine and closes at six from Monday to Saturday.' 'd2|A bike costs twelve pounds to hire for a day and five pounds for an hour.' 'd3|You can return a bike to any of the three stations in the city.' 'd4|If a bike breaks, call the number on the frame and we will fix it.' 'd5|Helmets are free with every hire but you must ask at the desk.' > faq.txt"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the six test questions
docker exec jupyter sh -c "printf '%s\n' 'When does the shop open?|nine' 'What is the cost of a day?|twelve pounds' 'Where can I return a bike?|three stations' 'What do I do if my bike breaks?|call the number' 'Do I get a helmet?|free' 'What time is closing?|six' > questions.txt"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the scoring script
docker exec jupyter sh -c "printf '%s\n' 'import sys' '' 'SKIP = [\"the\", \"is\", \"a\", \"an\", \"of\", \"for\", \"do\", \"i\", \"what\", \"when\", \"does\", \"how\", \"my\", \"if\", \"can\", \"are\", \"to\", \"where\", \"get\"]' '' '' 'def words(text):' ' for mark in \".,?\":' ' text = text.replace(mark, \"\")' ' return text.lower().split()' '' '' 'docs = []' 'for line in open(\"faq.txt\"):' ' docs.append(line.strip().split(\"|\")[1])' 'skip = []' 'if sys.argv[1] == \"skip\":' ' skip = SKIP' 'hit = 0' 'wrong = 0' 'none = 0' 'for line in open(\"questions.txt\"):' ' question, fact = line.strip().split(\"|\")' ' asked = []' ' for word in words(question):' ' if word not in skip:' > steps/ask.py"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the rest of the scoring script
docker exec jupyter sh -c "printf '%s\n' ' asked.append(word)' ' best = \"\"' ' best_shared = 0' ' for text in docs:' ' shared = 0' ' for word in asked:' ' if word in words(text):' ' shared = shared + 1' ' if shared > best_shared:' ' best = text' ' best_shared = shared' ' if best == \"\":' ' none = none + 1' ' print(\"none:\", question)' ' elif fact in best.lower():' ' hit = hit + 1' ' print(\"hit:\", question)' ' else:' ' wrong = wrong + 1' ' print(\"wrong:\", question)' 'print(\"hits\", hit, \"wrong\", wrong, \"none\", none, \"hit rate\", round(hit / (hit + wrong + none), 2))' >> steps/ask.py"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Score the plain matching
docker exec jupyter python3 steps/ask.py plainShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Fix one thing and score again
Two questions went to the wrong paragraph because of small words like "the" and "a". Take such stop words out of the question before counting, and score again.
One change is made at a time. If you changed three things at once you could not say which one helped.
Score again, skipping the stop words
docker exec jupyter python3 steps/ask.py skipShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a hit rate of 0.67 shows the app is good. The rate went up, and both wrong answers became hits or no answers. But two questions still find nothing, and six questions are very few. A test set this small tells you where to look, not how good the app is.
Worked example: the two runs side by side
| Run | Hit | Wrong | No answer | Hit rate |
|---|---|---|---|---|
| Whole words, nothing skipped | 3 | 2 | 1 | 0.5 |
| Stop words skipped | 4 | 0 | 2 | 0.67 |
The cells students fill in are left blank in this preview.
Going further: add your own questions
Add four more questions to the test file, each with a fact. Include one the shop's five answers cannot answer at all.
Run both scores again. What should the right outcome be for the question with no answer in the shop's text? Does the app get it right?