AI comparison
Evals vs unit tests
Unit tests check that code does exactly what it should every time; evals score how well a model-backed feature does over many cases, because its outputs vary.
| Aspect | Evals | Unit tests |
|---|---|---|
| Result | A score, such as 46 of 50 | Pass or fail |
| Output checked | Varied text, judged by rules, references or a judge | An exact, known value |
| Speed and cost | Slower, may call a model | Fast and free |
| When it fails | Look at which cases got worse | A specific assertion broke |
Choose Evals when
- The output is generated text or a model decision
- You compare prompts, models or retrieval settings
- Good enough is a threshold, not a single answer
Choose Unit tests when
- The code is deterministic, such as parsing or validation
- You check the code around the model
- You want a fast check on every save
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.
Related terms
- Eval
- An eval is a repeatable test of an AI feature: a fixed set of inputs, a way to score each output, and a result you can compare between versions.
- Golden set
- A golden set is a fixed collection of inputs with agreed good answers, used as the trusted reference an AI feature is scored against.
- LLM as judge
- LLM as judge is using a language model to score another model's output against a rubric when a simple code check cannot decide.