AI glossary
Token
A token is the piece of text a language model reads and writes in: often part of a word, a whole short word or a punctuation mark.
Before a model sees your prompt, a tokeniser cuts it into tokens and turns each into a number. In English a token is roughly four characters, so "unbelievable" may be three tokens while "cat" is one.
Tokens are what you pay for and what limits you: an API charges per token in and per token out, and the context window is measured in tokens. Counting them before a call is how you keep a cost and a prompt in bounds.
Other languages and code often take more tokens per word than English, so a budget worked out on English text can run short on Japanese or on a long stack trace.
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.