How text is split into tokens

Tokenisation (also spelt tokenization) is splitting a piece of text into small pieces called tokens. It is the first step of natural language processing, the part of computing that works with human language.

A program cannot read a sentence the way you do. Before it can count, compare or search anything, it has to decide where one piece ends and the next begins.

A token is one piece of text that a program treats as a single unit. You choose what counts as one piece. The usual choice for a simple program is a word.

Splitting on spaces is the easiest rule. Python does it with split(). The result is a list, and each item in the list is a token.

sentence = "The shop opens at nine and closes at six"
tokens = sentence.split()
print(tokens)
print(len(tokens), "tokens")
# Add "on Sunday" to the sentence. How many tokens now?

docker exec jupyter head -n 1 hires.csv

The first line of hires.csv has seven names joined by commas. A space rule would find one token. A comma rule finds seven: date, month, weekday, weather, temp_c, rain_mm and hires.

The same text gives different tokens under a different rule. So you always say which rule you used.

Split "The shop opens at nine." on spaces and the last token is nine. with the full stop stuck to it. A program sees nine. and nine as two different tokens.

Cleaning fixes that before the split. Change everything to lower case, so "The" and "the" match. Remove the punctuation marks, such as the full stop and the comma.

docker exec jupyter python3 -c "print('The shop opens at nine.'.lower().split())"

text = "Late again, the bus is full!"
text = text.lower()
for mark in [".", ",", "!", "?"]:
    text = text.replace(mark, "")
tokens = text.split()
print(tokens)
# Add a question mark to the text. Is it still removed?

Some words appear in almost every sentence: the, a, at, and, is. These are stop words. They tell you little about what a text is about.

Many programs remove them after splitting, to keep the tokens that matter. Whether to remove them is a choice. For some tasks, such as answering a question, they matter, so keep them.

A word is not the only unit. A token can be one character, or a part of a word. Large language models use subwords: common words stay whole and rare words are cut into pieces.

The vocabulary is the list of different tokens. A word you never saw is an unknown token, and character tokens avoid that problem.

docker exec jupyter python3 -c "print(list('shop'), len('shop'))"

Take a sentence of your own about your week and tokenise it by hand, then with the code above. Do the two lists match?

Next, find a sentence where removing stop words changes the meaning, such as "not on time". Explain what the program would lose.