Want more reliable AI results without scaling human review? Simon Willison’s recent write-up on “Judgement” highlights a simple, powerful pattern: use an LLM as a judge to score and compare model outputs against a clear rubric.
This isn’t magic; it’s structured evaluation. With the right safeguards, a judge model can catch errors, reward better answers, and make your pipeline measurably stronger.
What is the LLM‑as‑Judge pattern?
An LLM is prompted to critique or score model outputs using explicit criteria. You can grade a single answer, compare two answers, or check for violations (like hallucinations or policy issues).
- Single-answer grading: Score one output against a rubric (e.g., 1–5 for factuality, clarity, citations).
- Pairwise comparison: Present two answers in random order and ask the judge to pick a winner with a short rationale.
- Rule checking: Ask the judge to flag policy, safety, or style violations and suggest fixes.
Why it works (and where it fails)
Judge LLMs can correlate surprisingly well with human preference judgments, especially for pairwise comparisons and clear rubrics. Public benchmarks like MT-Bench and AlpacaEval popularized this approach for scalable evaluation.
- Strengths: Fast, cheap, consistent; scalable beyond small human panels; great for regression testing.
- Limits: Susceptible to verbosity and position bias; may reflect training-data biases; can be gamed by clever prompting.
- Mitigations: Randomize answer order, hide system prompts, require a rationale, aggregate multiple judges, and spot-check with humans.
Quick start: add a judge to your pipeline
- Define a rubric with 3–5 precise criteria (e.g., Factuality, Completeness, Harm Avoidance, Evidence, Clarity). Include 1–2 sentence definitions and examples.
- Use pairwise tournaments: Compare candidate outputs in randomized order; pick winners until you have a champion.
- Separate models: Prefer a different model (or version) for judging than for generation to reduce shared blind spots.
- Ask for confidence + short rationale: “Pick A or B, explain why in 1–2 sentences, and give confidence 0–100.”
- Aggregate: Majority vote across 3+ independent judges or different seeds; break ties with human review.
- Harden against leakage: Keep references and gold answers hidden; avoid leaking system prompts; use minimal context for judging.
- Track metrics: Store rubrics, prompts, scores, rationales, and seeds. Watch win rates and failure modes over time.
Rubric starters you can adapt
- Factuality: Claims grounded in provided sources; no contradictions.
- Completeness: Addresses all parts of the question; no major gaps.
- Evidence: Cites sources precisely; avoids vague attributions.
- Clarity & Structure: Concise, logically organized, free of jargon.
- Safety & Policy: No harmful or disallowed content; mitigations noted.
Where this fits
- RAG QA: Judge answers against retrieved passages.
- Customer support: Grade macro suggestions before sending.
- Code assistants: Compare alternative patches or explanations.
- Content teams: Score outlines and drafts for tone and accuracy.
- Safety: Flag risky outputs before they reach users.
Sources and further reading
- Simon Willison: Judgement (blog post) – https://simonwillison.net/2026/Jul/3/judgement
- MT-Bench by LMSYS: LLM-as-a-judge for instruction-following – https://lmsys.org/blog/2023-06-26-mt-bench/
- AlpacaEval 2.0 (Tatsu Lab): Automated pairwise evaluation at scale – https://tatsu-lab.github.io/alpaca_eval/
Takeaway
Turn LLMs into evaluators, not just generators: write a crisp rubric, use pairwise judging with randomization, aggregate votes, and keep humans in the loop.
Enjoy this? Subscribe to The AI Nuggets for sharp, practical takes: https://theainuggets.com/newsletter
