How does a chatbot know your college's rules?
A college puts a help chatbot on its website. A language model has never read the college's handbook, so it cannot know the library rules. The program finds the right paragraphs first and sends them with the question. That is retrieval-augmented generation, or RAG. The model here is a stand-in that always gives the same answer, so you can check every result.
By the end of this lesson
Note
I can find the passages that answer a question and put them in a prompt with their sources.
Key words
- Chunk: a short piece of a longer document.
- Embedding: a list of numbers standing for a text.
- Vector store: chunks kept with their embeddings, for searching.
- Similarity: how close two embeddings point, from 1 down.
- Recall@k: how often the right source is in the top k results.
Starter: what you already know
Note
Answer in your head. 1. Which part of a chat request sets the rules? 2. Does a model remember earlier requests? 3. What does a search engine give back for a query? Answers: the system message; no, only what each request sends; a ranked list of pages.
Before you start
Note
Press Start the lab on the bar at the top of this lesson and wait until it says it is running. There are two containers. model is the stand-in, answering at http://model:4000/v1. tools has the lab's Python, five short documents in docs/ and scripts in steps/, and docker exec tools runs one command inside it. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.
Documents are cut into chunks
A whole document is too long to send with every question. So it is cut into chunks of a few words, each overlapping the last. The library page has 63 words. Chunks of 40 words with 10 overlapping give 2 chunks. Chunks of 8 give 8, and the fact "eight books" is now apart from "three weeks".
docker exec tools python3 steps/chunk.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
An embedding turns text into numbers
The first command asks the stand-in for 8 numbers for one question. The script asks for 128 each, always of length 1.0. Two texts sharing "borrow" and "books" score 0.866. "Laptop" against "notebook computer" scores 0.0. The stand-in hashes words, not meaning. A real embedding model places texts by meaning, so those two would be close.
curl -s -w "\n" http://model:4000/v1/embeddings -H "Content-Type: application/json" -d '{"model":"lab","input":"How many books can I borrow?","dimensions":8}'Shown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec tools python3 steps/embed.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Check
Tip
Two texts mean the same but share no words. What does the stand-in score them, and what would a real embedding model do? Answer: 0.0, because it only counts shared words. A real model would score them close, because it compares meaning.
A vector store finds the closest chunks
The store keeps 8 chunks from 5 documents, each with its numbers. A question is turned into numbers too, and every chunk gets a similarity score. The top three come back with their source. "How many books can I borrow?" finds library.txt first, scoring 0.333. A real system uses a vector database to search millions quickly.
docker exec tools python3 steps/search.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Found passages go in the prompt, with their source
The script prints the prompt: each passage starts with its source in square brackets, then the question. The answer names [library.txt], so a reader can check it. With no sources, the stand-in says it has no answer. With 8-word chunks, the passage with "three weeks" is missing, so it has no answer again.
docker exec tools python3 steps/answer.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
A common mistake
Warning
Many people think RAG teaches the model your documents. It does not: the model learns nothing and keeps nothing. It only reads the passages sent with that one question. If search misses the right passage, the answer is wrong or missing, however good the model is.
Recall@k measures the search
Eight test questions each have a known right source. Recall@k is the share whose source is in the top k. With 40-word chunks, recall@1 is 0.75 and recall@3 is 0.875. The notebook question shares no words with it.txt, so it is never found. Small chunks score higher, yet the last step showed they lose the fact. Measure the answer too.
docker exec tools python3 steps/recall.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
A document can carry hidden instructions
Only try this on data that is yours. notice.txt tells the assistant to say the exams are cancelled. Search finds it for an exam question, and the stand-in obeys it. Setting aside passages that give orders brings back the right answer. Real systems also treat passages as data, give the model no power to act, and check sources.
docker exec tools cat docs/notice.txtShown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec tools python3 steps/inject.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Exit check
Tip
Answer without looking back. 1. Why cut documents into chunks? 2. What does the stand-in's embedding compare? 3. Why give each passage its source? 4. What does recall@3 measure? 5. Is text found by search safe to obey? Answers: to send only what is needed; shared words; so the answer can be checked; how often the right source is in the top three; no, treat it as data.
For teachers
Tip
This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is two services of one Python image: a stand-in model with hashed embeddings, and a Python with a small vector store, five documents and short scripts. No real model is called, so it costs nothing and every output is the same. Key learning points: 1. Documents are cut into chunks and turned into embeddings. 2. Search returns the chunks closest to a question. 3. Found passages go in the prompt with their sources. 4. Recall@k measures the search, and found text is data, never orders. Common mistake: thinking RAG teaches the model. Equipment: the lab and a browser. Tip: ask students to predict which document each question finds before they press Run. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.