AI glossary
Inference
Inference is running a trained model to produce an output from an input, as opposed to training it, and it is what every API call or local run does.
For a language model, inference has two parts: reading the whole prompt at once, then generating the reply one token at a time. The first sets how long until the first word, the second how fast the rest arrives.
Inference cost grows with model size and with tokens in and out, which is why a smaller model, a shorter prompt or a cache can cut a bill sharply.
Training happens once; inference happens on every request. For most products it is inference that sets the running cost.
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.