AI glossary
Latency
Latency is how long a user waits for a response; for a language model it is usually split into time to first token and total time.
Time to first token is the wait before anything appears. Total time also includes generating every token of the reply, so a long answer takes longer even on a fast model.
Latency adds up across steps. A RAG answer that embeds, searches, reranks and then calls the model pays for each, and a trace shows which step is slow.
Measure percentiles, not averages. The slowest one in twenty requests is what users remember.
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.