# Ground-Truthing Your LLM Judge: Does Your Eval Actually Track Reality? > LLM-as-judge is the default eval now — but a judge is just another unreliable model. Here is how to check whether its verdicts correlate with real outcomes, and catch the leniency bias most judges have. Published: 2026-07-21 Tags: Evaluation, Agents Canonical: https://alexmlewis.com/writing/ground-truthing-llm-judges --- LLM-as-judge has quietly become the default way to evaluate AI systems. You can't hand-grade thousands of agent runs, so you ask a model: *is this output good?* It scales, it's cheap, and it feels rigorous — you have numbers now. Here's the problem nobody talks about: **the judge is just another unreliable model.** You added an LLM to grade your LLM, and then you started trusting the grades. Who evaluates the evaluator? The uncomfortable answer, most of the time, is *nobody*. Teams tune prompts and ship models based on a judge's pass rate without ever checking whether "the judge said pass" has anything to do with "this actually worked." That's not evaluation. That's a second opinion from someone with the same blind spots. ## The move: cross-tab the judge against ground truth You almost always have a ground-truth signal already — you're just not using it to grade the grader. The signal is whatever *actually happened* downstream: - For generated code: did it merge? did it pass CI? did a human approve it? - For a support answer: did it resolve the ticket, or did the user escalate? - For a retrieval answer: did the user click through, accept, or retry? The technique is embarrassingly simple, which is probably why it gets skipped: **build a confusion matrix between the judge's verdict and the real outcome.** | | Actually good | Actually bad | | ------------------- | ------------- | ------------ | | **Judge said pass** | true pass | *false pass* | | **Judge said fail** | *false fail* | true fail | Now your judge has a precision and recall against reality, like any classifier. And now you can see the thing that was invisible before: the shape of its mistakes. ## What I found: judges are lenient In my own work — an LLM judge scoring AI-generated changes, cross-tabbed against whether those changes actually merged — the dominant error wasn't random noise. It was **systematic leniency**: the judge waved through work that didn't hold up downstream far more often than it wrongly rejected good work. The false-pass cell was the heavy one. That makes sense if you think about how these judges are prompted. "Rate this response" with a friendly rubric optimizes for plausibility, and plausible-but-wrong is exactly the failure mode LLMs are best at producing. A judge reading a confident, well-structured answer is inclined to bless it — the same way a human reviewer rubber-stamps a clean-looking PR that turns out to be broken. Leniency is the dangerous direction, too. A judge that's too harsh wastes your time. A judge that's too lenient **ships the failures you built the eval to catch**, while giving you a green dashboard. ## Fixing it: guards for the crisp parts, the judge for the fuzzy part Once you can see the disagreement, two things help. **1. Take work away from the judge.** A lot of what people ask an LLM judge to check is deterministic: is the output valid JSON? does it contain the required fields? did the code compile? did the tests run? You don't need a probabilistic grader for any of that — you need an `if` statement. Put **deterministic verdict guards** in front of the judge so it only adjudicates the genuinely fuzzy question ("is this *good*?"), never the mechanical ones. Every check you can make deterministic is a check that can't drift. **2. Recalibrate the judge from the disagreement cells.** The false-pass and false-fail examples are the most valuable data you have — they're literally the cases where your instrument is wrong. Feed them back: tighten the rubric around the specific ways it's being fooled, add them as few-shot counter-examples, or split a vague criterion into concrete ones. Then re-run the cross-tab and watch the false-pass cell shrink. The point isn't a single before/after number — it's that you now have a loop. The judge stops being a fixed oracle you hope is right and becomes an instrument you can point at reality and tighten. ## The tradeoffs (because there always are some) - **Ground truth is often delayed and noisy.** A merge isn't a perfect label — good changes get reverted, bad ones slip through review. Treat the outcome as a strong signal, not gospel; you're looking for *systematic* divergence, not litigating individual cases. - **Not every task has a clean outcome.** Some outputs never get a downstream verdict. For those, you're stuck with the judge — which is exactly why you want to calibrate it on the tasks that *do* have ground truth and carry that trust over carefully. - **You can overfit the judge to the guard.** If you recalibrate too aggressively on one batch of disagreements, you can teach it that batch instead of the general rule. Hold out a fresh slice of outcomes to check. ## The actual takeaway Your eval is a measurement instrument, and instruments need calibration against reality — otherwise you're just measuring your own assumptions with extra steps. An LLM judge that nobody has cross- checked against outcomes isn't a quality gate; it's a confidence generator. So before you trust the next pass rate: pull the outcomes you already have, build the four-cell table, and look at how big the false-pass corner is. It's usually bigger than you'd like — and that's the most useful thing your eval has told you all quarter.