AI glossary
LLM as judge
LLM as judge is using a language model to score another model's output against a rubric when a simple code check cannot decide.
You give the judge the input, the output and a short rubric such as "Does the answer use only facts from the source? Reply pass or fail with a reason." It then scores many outputs quickly.
Judges have known biases: they favour longer answers, the first of two options, and text in their own style. Fixed rubrics, pass or fail rather than 1 to 10, and swapping the order all help.
Check the judge itself. Score a sample by hand and compare; a judge that disagrees with people often is measuring the wrong thing.
Learn it hands-on
These free lessons teach it with commands to run in a lab and checked tasks. Read them whole with no account.