Jeremy MacDonald’s argument that testing is what makes scale possible raises an obvious follow-up: how do you know the AI grader is any good? Jeremy didn’t cover this in his BrightonSEO talk, so this post adds published guidance on the topic. (Part five of my series. Catch up with the RISE eval and the feedback loop.)
The short version from one source: use LLM judges for volume and regression checks, and humans to define good and keep the judge calibrated. Or, as it puts it, the judge scales your standard and humans set it.
Known biases to design around
Sources describing LLM-as-judge setups commonly list the same weaknesses:
- Position bias. In pairwise comparisons, judges can favor whichever answer is listed first. One summary says up to 75% of cases, so randomize order and average across both orders.
- Verbosity bias. Judges tend to prefer longer answers. Add a rubric line that penalizes unnecessary length.
- Self-enhancement bias. A judge can favor outputs from its own model family. The same source reports roughly a 10% higher win rate for GPT-4 on its own outputs and about 25% for Claude-v1, from older studies, so use a judge from a different family than the generator where you can.
- Shared blind spots. High agreement between judges doesn’t prove they match human judgment; a panel can be consistently wrong together.
The agreement research is encouraging. In Zheng et al. (2023), “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, GPT-4 judges reached more than 80% agreement with human preferences, roughly matching human-to-human agreement. That was for general chat quality, not marketing work, so test it on yours.
A calibration workflow
- Label a frozen set. Have humans score 150 to 300 real inputs using the same rubric the judge will use.
- Check humans agree with each other first. A Cohen’s kappa of roughly 0.7 or higher is the benchmark the source suggests.
- Measure the judge against the humans. If it lags the human-to-human benchmark, tighten the rubric, add anchored examples or change the judge model.
- Randomize and normalize. Average scores across both presentation orders and strip length cues.
- Keep sampling. Send 5 to 10% of judged items to humans on an ongoing basis.
- Re-run quarterly and after any model or rubric change.
Why you calibrate per criterion
A worked example in the Koji source shows why. The judge kept up with humans on simple yes/no criteria, but fell well short on a subjective 1-to-5 usefulness scale:
Another guide from FutureAGI recommends a gold set of 200 to 500 labeled items, alerting when judge-to-human agreement falls below about 0.6, and versioning your rubric like production code. Treat all of these thresholds as starting points from vendor blogs, not standards.
Where each one fits
| Use an AI judge for | Keep humans for |
|---|---|
| Regression checks on every change | Defining quality for something new |
| Grounding and factual consistency, with a human audit | Brand voice and tone calls |
| Scoring thousands of outputs | Ship or no-ship decisions on sensitive content |
| Comparing prompt variants with large differences | Understanding why outputs fail |
What this means for Jeremy’s workflow
The keyword-research eval from part three is a good candidate for an AI gate because the criteria are fairly concrete. A brand-fit score is more subjective, so audit it more often. The principle underneath is his own: test the work at scale, and test the tester too.
It’s the same reason I argued for human eyes on AI-generated audits: AI scales the work, but humans still have to set the standard.
Want help setting up evals you can actually trust? Get in touch.
Leave a Reply