Even the yardstick needed calibrating

Category: AI Engineering/Benchmarking

Two well-behaved judges, the same 249 answers, a 24-point disagreement, and what that means for every AI accuracy number you'll ever be shown.

Post 1 used a test to pick a model, and post 2 showed how the test was built: a 273-question battery, generated from the documents themselves, graded by a second language model. Neat. Rigorous. Slightly smug.

This post is about the moment the measuring tape turned out to be made of elastic.

Rule zero: measure the noise first

Before trusting any improvement, I ran the identical setup twice. Same code, same questions, same sources, same model. In a perfect world the results would be identical.

They weren't. 23 of 273 results changed between the two runs: 10 questions went from fail to pass, 13 went the other way. Nothing had changed except luck. Language models sample their words with a little randomness, and so does the model grading them.

That number, 23 flips, became the bar every change had to clear. And the very first change I tested fell flat on its face against it.

The improvement that wasn't

The battery had found a real weakness: the model played along with leading questions ("Why did I sell my sailboat?" when there was no sailboat), and it fumbled yes/no questions. So I added two rules to the prompt, one for each problem, and ran the battery again.

Result: 34 questions flipped, 16 for the better and 18 for the worse. Net: −2.

Compared with the noise floor of 23 flips, that's indistinguishable from doing nothing. Without the A/A run I would have looked at the 16 fixed questions, felt clever, and shipped it. With it, the honest conclusion was "no measurable effect". (The targeted questions did move the right way: leading traps refused 3 of 5 → 4 of 5. That's a hint, not evidence.)

Worse, one of the two rules turned out to have a side effect that only showed up later: the yes/no instruction made the model start a quarter of all answers with "Yes,", including "Yes, the kayak is number 61". Rules leak. Now they're only added for the question type they're meant for.

The judge with a mood

Looking closer at the flips, something odd showed up. Several "broken" answers were almost word-for-word identical to the version that had passed. The answer hadn't changed. The verdict had.

The judge, a separate language model grading each answer against the reference, was itself sampling with a little randomness. A judge with moods. Setting its randomness to zero (temperature 0, fixed seed) fixed that completely: the same 22 answers graded twice gave 22 of 22 identical verdicts, and agreement with my own hand ratings stayed the same.

Lesson one: if your grader can change its mind, part of your "noise" is the grader.

Two judges walk into a benchmark

The bigger-model comparison in post 1 needed a new judge, because a model can't fairly grade its own answers. I calibrated a second judge, a different and larger model, against my hand ratings. It agreed with me on 20 of 22 answers. The first judge had managed 19 of 22. Both looked trustworthy.

Then I graded the same 249 answers with both:

Same answers. 24 percentage points apart. They disagreed on 61 answers, and in 60 of those the second judge was the generous one.

So who was right? I settled 14 of the disagreements by hand. Each judge was right exactly 7 times.

• The strict judge marked correct answers as partial, for instance "May 19" without the year.

• The lenient judge passed a refusal ("the sources don't contain this") as a correct answer, approved a list with wrong extra items, and accepted the wrong meeting time.

The truth sat roughly in the middle: about 67 % fully correct, give or take a dozen points.

The uncomfortable part is why calibration didn't catch it. My 22 hand-rated answers were a random sample, and random samples are mostly easy cases where any sensible judge agrees. The judges differed on the borderline answers: partial, ambiguous, almost-right. That's exactly where a calibration set needs to be dense, and mine wasn't.

Lesson two: calibrate judges on the hard cases, not a random sample. The disagreements between two judges are a ready-made list of those hard cases.

So which numbers can you trust?

This didn't sink the project's conclusions. It sorted them into two piles.

Comparisons with one judge: trustworthy. The small-versus-big model result in post 1 was graded by one judge on both sides, and a judge-free measure (literal answer words) pointed the same way. A judge's strictness shifts both models equally.

Absolute numbers: report a range. "The assistant is 67 % correct" should really read "between 55 % and 79 % depending on how strict you are, probably around 67". I now quote accuracy as a strict-to-lenient range, and stopped saying "80 % correct" without one.

The quieter calibration errors

Judges weren't the only thing that needed adjusting. Four smaller cases, each of which would have skewed a decision:

1. Refusals counted as wrong answers. My first "precision of given answers" counted "I can't find that in the sources" as a wrong answer. A refusal isn't wrong; it's an abstention. Corrected, the system's precision went up by several points on the same data. The system hadn't improved, only the arithmetic.

2. The "I don't know" threshold was calibrated on the wrong material. The threshold for when the assistant should abstain was tuned on my email archive. On a different kind of text, technical documentation, the same threshold refused a question whose answer was right there, because the reranker scored documentation lower than chatty email. Thresholds are calibrated per knowledge base, not once.

3. Wrong references in the answer key. About one generated question in five had to be dropped. A few that survived still had wrong expected answers ("9 years", when the source clearly said "just over ten"). The judge dutifully marked the correct answer wrong. Garbage in, gospel out.

4. A runaway grader. The "thinking" judge once kept reasoning until it hit a ten-minute timeout, and took a whole benchmark run down with it. The answers hadn't been saved yet. Now answers are saved before grading, a failed grading gives "no verdict" instead of a crash, and the judge's thinking has a token cap.

Two kinds of instruments

The most useful idea to come out of all this was splitting the measurements into two kinds.

Exact instruments. The search step is deterministic: same question, same sources, every time. A tool that traces where the right source gets lost gives identical numbers on every run. So a change that fixes 12 questions and breaks 1 has really done that. No noise floor needed, and a run takes minutes.

Noisy instruments. Anything involving a language model answering or grading varies from run to run. Those need the noise floor, question-by-question comparison, a calibrated judge and a range on absolute numbers. A run takes hours.

So: tune what you can measure exactly with the exact instrument, and save the noisy one for what's left. Search got tuned in minutes; answer quality got judged in hours, and much more humbly.

Takeaways

1. Run an A/A test first. Run the same setup twice. Your noise floor is the number of results that change.

2. Zero the judge's randomness. A grader with moods adds noise you'll mistake for effects.

3. Calibrate judges on the hard cases. Random samples are mostly easy; judges differ on the borderline ones.

4. Compare with one judge; report absolute numbers as a range. Strict and lenient judges can be 20+ points apart on the same answers.

5. Check the arithmetic. Is a refusal counted as wrong? Is the threshold tuned on the right material? Is the answer key itself right?

6. Separate exact instruments from noisy ones, and use the exact one wherever you can.

Or in one sentence: even the yardstick needed calibrating: running the identical test twice changed 23 of 273 results, and two AI judges disagreed by over 20 percentage points, so without careful measurement we would have drawn the wrong conclusions.

Next: "This looks like a question we've answered before". Why serving a stored answer to a similar question went wrong 66 times out of 66, and the three checks that fixed it.