LLM as a judge
Also called: model-graded eval, auto-rater, AI grading AI
Using one language model to score another model's output against a rubric — cheaper and faster than human rating, so it can run across thousands of cases. It approximates human judgment; it does not replace it.
It's how teams evaluate at a scale humans can't sustain. But the judge inherits the same failure modes as any model: it can be biased toward longer or more confident answers, fooled by fluent nonsense, and swayed by wording. A judge is only trustworthy once you've checked its verdicts against real human ratings on a sample.
Ground truth. A model's score is an estimate to be validated against humans, not the final word — and never let the same model both write and grade the answer without a human spot-check.
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1U.S. National Institute of Standards and Technology · 2023-01-26
First-pass citations, limited to primary sources; a reviewer will broaden and verify these before this entry leaves draft.