Shipping an LLM feature? Skip fragile benchmarks and run SME evals—fast, focused reviews by subject‑matter experts that reveal product‑level quality and risks.
In a timely post, Simon Willison highlights why expert evaluations often beat leaderboard‑style metrics when you need to know, “Is this good enough for our users?”
What are SME evals?
SME (subject‑matter expert) evaluations are structured human reviews of model outputs using a simple rubric. They measure usefulness, accuracy, safety, and fit for your domain.
Unlike synthetic or leaderboard benchmarks, SME evals reflect your real data, tasks, and risk tolerance—so they’re better signals for go/no‑go decisions.
When to use them
- Before launch: sanity‑check quality and safety on real tasks
- After model swaps: compare regression vs. improvement in user‑visible outcomes
- For sensitive workflows: confirm correctness and citations where stakes are high
1‑hour SME eval quickstart
- Define the task and success: one sentence, one owner, one decision date.
- Assemble a small sample: 20–30 real prompts/cases covering typical and tricky edge cases.
- Create a 3–5 point rubric: clarity, factuality, actionability, and any domain criticals (e.g., citations).
- Recruit 2–3 experts: different backgrounds if possible; blind them to the model identity.
- Rate independently: 1–5 per criterion + overall pass/fail with justification notes.
- Measure agreement: aim for substantial agreement (e.g., Cohen’s κ ≈ 0.6+ as a heuristic).
- Adjudicate: discuss disagreements, update rubric, and re‑score a small subset if needed.
- Decide and document: 1‑page memo with examples, scores, risks, and a ship/no‑ship call.
Copy‑paste rubric (starter)
- Accuracy (1–5): Are facts and calculations correct? Evidence/citations provided when required.
- Relevance (1–5): Directly answers the prompt without fluff or hallucinated scope.
- Actionability (1–5): Clear, stepwise, and usable by the target persona.
- Safety/Compliance (1–5): No risky, private, or policy‑violating content.
- Overall (Pass/Borderline/Fail): Would you ship this answer to a real user as‑is?
What to report
- Mean scores per criterion + overall pass rate
- Inter‑rater agreement (e.g., Cohen’s κ) and number of raters
- Representative examples: 3 wins, 3 fails, and 3 borderlines with notes
- Known risks, mitigations, and a crisp decision (ship, fix, or block)
Common pitfalls (and fixes)
- Rubric creep: keep 3–5 criteria; trim anything not tied to the decision.
- Sampling bias: include edge cases and “known hard” scenarios from support logs.
- Uncalibrated raters: run a 5‑case dry‑run to align on scoring.
- Model peeking: blind raters to models to avoid brand bias.
- Over‑indexing on averages: always pair scores with concrete examples.
Sources
Further reading: Simon Willison on SME evals; complementary background on eval frameworks from Stanford CRFM’s HELM.
Takeaway
SME evals turn fuzzy model claims into clear, product‑ready evidence. With a tiny sample and a tight rubric, you can decide faster—and ship safer.
Enjoy this nugget? Subscribe to our free newsletter for weekly, practical AI guides: theainuggets.com/newsletter.
