Evidence & Null Results
Our LLM-as-a-judge returned a perfect score 92% of the time.
That was the problem.
LLM as a judge is an evaluation method where a language model scores another model's output against a set of criteria, standing in for human review. It is fast, cheap, and scales to every run your system makes. It also has a failure mode nobody warns you about, and it is not noise. It is saturation. We know because it happened to us, and we measured it.
What we found
Our orchestration platform runs a quality score on every agent execution: a conformance ratio, the fraction of acceptance criteria a model judge marks satisfied. In one day of measurement across a full day of executions, that judge returned exactly 1.00 on roughly 92% of non-failed runs. Three of our four task types showed literally zero variance across their replicates. Not low variance. Zero.
The score distribution proved the mechanism rather than the model: the non-perfect values landed on clean fractions of the criteria count, exactly what a ratio of satisfied criteria produces. The judge was doing its job correctly. Its job just has a ceiling, and on well-specified tasks the honest answer to "did you do what was asked" is yes, all of it, every time. About 75% of all our runs returned the same single value.
A saturated instrument does not announce itself. It hands you clean statistics about a reading it can no longer take.
Why saturation is invisible from inside
Here is the part that should worry anyone using LLM evaluation metrics to make decisions. From inside the data, a saturated judge looks precise. Our pooled within-task standard deviation was σ = 0.049, a beautifully tight number. We fed it into a standard power calculation for a pre-registered experiment, and the formula cheerfully told us a single-digit number of matched pairs would suffice. The arithmetic was correct. The meaning was empty. σ was small not because the system was precise but because the measure had nowhere to move.
The score that couldn't tell truth from fabrication
The sharpest single result: in a matched-pair comparison, one output fabricated mechanisms that didn't exist on the platform, and the paired output stuck exactly to what the source material said. The judge scored both 1.00.
Both outputs satisfied the acceptance criteria as written. Conformance answers "did you do what was asked." It is structurally blind to "is what you said true." No prompt tweak fixes that, because it isn't a prompting bug. It is a dimension the instrument doesn't have.
How we found out: an experiment on ourselves
This didn't come from a benchmark paper. It came from a pre-registered, matched-pair harm test we ran on our own platform, in one day: byte-identical container clones, a pinned engine version verified before and after the batch, intent-to-treat analysis across matched pairs on four task types.
The question was whether a specific platform feature degraded output quality. Answer: no harm detected. Combined effect −0.01, 95% CI [−0.045, +0.025], with zero contamination in the control arm.
One scope limit, and it's unremovable: because the control arm sat at the ceiling almost without exception, improvement was undetectable by construction. This study can never be cited as evidence the feature helps, only that it doesn't hurt. We're stating that limit ourselves because it's the honest reading, and because a ceiling that hides improvement is exactly the instrument failure this article is about.
We publish what's working and what isn't. This one is a flop with a useful shape.
The fix is structural, not prompt-tuning
The industry reflex when a judge misbehaves is to rewrite its prompt or swap in a bigger model. Our judge was already model-based and already correct at its one job. The fix is to change what the instrument measures, not how nicely we ask.
A vector, never a single number. Four independent dimensions: conformance (did the output do what was asked; unchanged, it's good at this), groundedness (count of unsupported assertions, 0 to n; a count doesn't saturate), minimality (how much was emitted versus how much was necessary), and collateral (how much was touched outside the claimed scope, kept as a magnitude instead of a pass/fail).
Never averaged. Averaging the vector back into one number re-saturates it. You rebuild the ceiling you just escaped.
Versioned per row. Every scored execution carries the version of the instrument that scored it. The moment a judge improves, a 0.84 from June and a 0.84 from August stop meaning the same thing, with nothing in the data to show it. Version stamps let the instrument evolve without silently corrupting every longitudinal comparison. And instrument changes land between studies, never mid-batch, because a mid-batch instrument change is indistinguishable from a treatment effect.
The circularity rule. A judge must never consume the thing it evaluates. If the judge reads the same feature you're testing, it recognizes and rewards feature-shaped output, inflating the very effect it should detect. That's not noise; it's unfalsifiable from inside the data. Our resolution: same code, two frozen configurations. The judge-as-product gets every enrichment; the judge-as-instrument gets a frozen prompt, a pinned model, and no access to any arm of the experiment.
Common questions
Is LLM-as-a-judge reliable?
Reliable at what it measures. A conformance-style judge answers "did the output meet the stated criteria" consistently. The risk isn't randomness. It's that on well-specified tasks the score pins to the ceiling and stops carrying information, while still looking statistically clean.
How do you detect judge saturation?
Look at the raw score distribution, not the summary statistics. Warning signs we hit: a large majority of runs on one exact value, task types with zero variance across replicates, and power calculations that return implausibly tiny sample sizes. Tight σ on a ceilinged instrument is a symptom, not a virtue.
What should replace a single quality score for AI agent evaluation?
Independent dimensions reported as a vector: conformance, groundedness, minimality, collateral in our case, with the instrument version stamped on every row and no cross-version comparison. And a separation rule: the evaluator never ingests the treatment it's evaluating.
Can you run controlled experiments on your own AI platform?
Yes, and it's the method that caught all of this: matched container clones, pinned versions, pre-registered analysis, intent-to-treat, run and analyzed same-day. The experiment took an afternoon. The instrument finding was worth more than the experimental answer.
This article reports measured results from a pre-registered study on our own platform. The study register, including the full mistake log, is our source of record.
Related writing
- The Checklists That Did Nothing — the null result this instrument finding came out of.
- The Median Trap — the controlled A/B on front-loaded instructions.
- The 85% Accuracy Trap — what per-job scores are really telling you.