The chatbot that made up a refund rule
A delivery firm puts a chatbot on its website. A customer asks how long a refund takes. The chatbot says 30 days, sounding sure.
The firm’s real rule is 14 days. The chatbot never saw that rule. It wrote what sounded likely.
The fix is not a bigger model. It is giving the model the firm’s own page before it answers.
By the end of this lesson
Note
I can explain how an app gives a language model the right facts.
Key words
| Word | What it means |
|---|---|
| Language model | a model that writes text by predicting the next words |
| Prompt | the text you send the model |
| Hallucination | a confident answer that is made up |
| Retrieval | finding the right passages in your own documents |
| Chunk | a short passage cut from a document |
| Vector | a list of numbers standing for a text’s meaning |
The cells students fill in are left blank in this preview.
A language model only knows its training text
A language model is a very large neural network trained on public text. It writes an answer by predicting likely next words.
It has never seen your firm’s policies, prices or customers. It also stops learning on the day training ends.
Asked about something it does not know, it often still answers. A confident, made-up answer is a hallucination.
What a model knows and does not
| Question | In its training text? | Likely result |
|---|---|---|
| What is a refund? | Yes, it is common | A good answer |
| How long do our refunds take? | No, it is our own rule | A guess, maybe wrong |
| Where is order 48213? | No, it is our live data | A made-up answer |
The cells students fill in are left blank in this preview.
Retrieval puts the facts in the prompt
The app does not change the model. It changes the prompt.
First it cuts the firm’s documents into short chunks. When a question arrives, it searches them for the best matches. That step is retrieval.
Then it sends the model the question and those chunks, with an instruction: answer only from these. This whole pattern is called retrieval-augmented generation, or RAG.
Find the best chunk
# The firm's help pages, cut into short chunks.
chunks = [
"refunds are paid within 14 days of the return arriving",
"parcels over 20 kg need a two person delivery",
"you can change the delivery address until the parcel is sent",
]
question = "how many days until my refunds are paid"
for chunk in chunks:
score = 0
for word in question.split():
if word in chunk.split():
score = score + 1
print(score, "shared words:", chunk)
# Change the question to "can I change the delivery address", then press Run.
# Which chunk scores highest now?
Vector search matches meaning, not words
Counting shared words has a weakness. "Money back" and "refund" mean the same, but share no words.
So real apps turn each chunk into a vector: a list of numbers standing for its meaning. Texts with similar meanings get similar numbers.
The question is turned into a vector too. The search then finds the chunks whose numbers are closest. The numbers are stored in a vector database.
A common mistake
Warning
Many people think an app teaches a model its facts by training it again. A RAG app usually does not. It finds the facts at the moment of the question and puts them in the prompt. So updating a document updates the answers straight away.