Why is text cleaned before searching?
Text processing for search turns raw text into clean tokens: lower case, no punctuation, no stop words and words cut back to a stem. Without it, "Bus", "bus." and "buses" would be three different words to a search system.
By the end of this lesson you will write a tokeniser that makes the same token from every form of a word.
By the end of this lesson
Note
I can turn raw text into good tokens.
Key words
| Word | What it means | Example |
|---|---|---|
| Token | one piece of text that is looked up in the index | bus |
| Normalisation | making different spellings of a word the same | Bus and bus |
| Stop word | a very common word that is left out | the |
| Stemming | cutting a word back to its stem | |
| Stem | the part of a word that stays the same | delay |
The cells students fill in are left blank in this preview.
What is wrong with raw text?
A computer compares strings exactly. "Bus" and "bus" are different, and so are "late," and "late". If the index holds one and the query uses the other, the document is not found.
So the same steps are applied to every document and every query. Then equal words become equal tokens.
Lower case and remove punctuation
text = "The Bus is LATE, again!"
text = text.lower()
for mark in [",", "!", ".", "?"]:
text = text.replace(mark, " ")
print(text.split())
# Put "bus-stop" in the text. Add "-" to the list of marks.
Look at the first line of a file
docker exec jupyter head -n 1 hires.csvShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Clean a line of text in one step
docker exec jupyter python3 -c "print('The Bus is LATE, again!'.lower().replace(',', ' ').replace('!', ' ').split())"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: what each step does
| Step | Text after the step |
|---|---|
| Raw text | The Bus-stop is BUSY! |
| Lower case | the bus-stop is busy! |
| Split at punctuation | the, bus, stop, is, busy |
| Drop stop words | bus, stop, busy |
The cells students fill in are left blank in this preview.
What are stop words?
Stop words are very common words, such as "the", "is" and "of". They are in nearly every document, so they do not help tell one from another. The idf of such a word is close to 0.
Dropping them makes the index smaller and the ranking cleaner. The list is a choice. A phrase such as "to be or not to be" would vanish.
What is stemming?
Stemming cuts a word back to a stem, so that "delay", "delays", "delayed" and "delaying" are one token. A search for one finds the others.
A simple stemmer only strips endings by rules, such as "ies" to "y", then "ing", then "ed", then a final "s". It does not know any English.
Worked example: a simple stemmer at work
| Word | Rule used | Stem |
|---|---|---|
| tickets | remove the final s | ticket |
| delayed | remove ed | delay |
| delaying | remove ing | delay |
| studies | ies becomes y | study |
| bus | too short to change | bus |
| news | remove the final s | new |
The cells students fill in are left blank in this preview.
A simple stemmer in Python
def stem(word):
if word.endswith("ies") and len(word) > 4:
return word[:-3] + "y"
if word.endswith("ing") and len(word) > 5:
return word[:-3]
if word.endswith("ed") and len(word) > 4:
return word[:-2]
if word.endswith("s") and not word.endswith("ss") and len(word) > 3:
return word[:-1]
return word
for word in ["tickets", "delayed", "delaying", "studies", "bus", "news"]:
print(word, stem(word))
# Try "running". Is the result a real word?
A common mistake
Warning
Many people think a stem is always a real word. A crude stemmer turns "news" into "new" and "running" into "runn". That is acceptable, because the stem only has to be the same for the words that should match, in the index and in the query.
Going further
Write one sentence about a bus journey in three forms: with capitals and a comma, then plain, then with the verb in another tense. Tokenise all three by hand and check that the tokens match.
Then find a word where the simple stemmer joins two words that should stay apart, and say what a search for one would wrongly find.