How to test an assistant on documents you've never read, and why the test needed testing too.
In the last post, a small model beat a bigger one. That only means something if the test was any good. This post is about the test: 273 questions, most of them written by a machine, graded by another machine, and kept honest by a few humbling surprises.
The problem: grading an exam you can't answer yourself
Company A wants an assistant that answers questions from their documentation. I don't know their documentation.
Neither, in full, does anyone working there. So how do you check whether the assistant's answers are right?
The obvious route is expensive and fragile: ask experts to write a few hundred questions with correct answers. That takes days. It also turns out to be error-prone. When I wrote expected answers for my own archive, the archive I do know, I got two of them wrong. One "trap" question I was sure had no answer ("what is my passport number?") turned out to be answerable: the number was in there, under a spelling my search didn't catch. Humans make lousy answer keys.
Turn the question around
The trick is to stop asking "what's the answer to this question?" and ask "which text did this question come from?" instead:
Pick a random passage from the documents.
Let a language model write a question that the passage answers, plus the exact words from the passage that a correct answer must contain.
Now you know where the answer lives and what it says, without anyone having read the documents.
It's the exam equivalent of writing the answer key first and the questions second. Teachers call that cheating. Engineers call it test generation.
Generators make mistakes, so every question passes mechanical checks before it's kept:
Check | Why |
|---|---|
The answer words must appear, word for word, in the passage | Otherwise the generator invented the answer |
No "the email", "the document", "this passage" in the question | A real user hasn't seen the passage |
Reworded questions must share few words with the passage | Otherwise it tests keyword search, not understanding |
Answer words that already appear in the question are dropped | Models repeat the question, which inflates the score |
"Trick" questions must name a detail that is not in the passage | Otherwise the premise isn't actually false |
About one generated question in five fails these checks. The worst category was trick questions: the generator kept writing questions that were simply true, which is a very polite way to fail at lying.
A battery, not a score
A single number such as "85 % correct" tells you nothing about what to fix. Psychologists solved this a century ago: an intelligence test is a battery of subtests, each measuring one ability, so a weak score points at one skill instead of the whole person.
An assistant like this has distinct abilities too, and each lives in a different part of the machinery. So every test question gets a type, in three families:
Can it find the source? (search)
Direct lookups that reuse the source's words.
Paraphrases with almost no words in common with the source.
Names and codes, which meaning-based search tends to blur.
Questions in English about Swedish sources, and the reverse.
The same question with typos, and in search-box style ("broken link fix dashboard").
Can it read what it found? (the language model)
Exact numbers, dates and amounts.
Yes/no questions, where "not" matters.
"What's true now", when older and newer sources disagree.
Questions that need several sources at once.
Does it know what it doesn't know? (honesty)
Questions about topics that simply aren't in the material.
General knowledge ("What's the capital of Australia?"): a FAQ assistant should decline, however tempting Canberra is.
Leading questions that assume something that never happened ("Why did I sell my sailboat?" when there was no sailboat).
False premises: a real event with one detail changed ("the 2024 conference" when it was 2023).
For a business FAQ, the third family matters most. A confident wrong answer costs more than "I don't know".
The typo and keyword variants are my favourite part. They're the same question in worse clothes, so a robust system should answer them just like the original. It's a robustness test you get for free.
The test lied to me (three times)
The battery found bugs in the system. More embarrassingly, it found bugs in itself first.
1. The yes-man problem. Of 16 generated yes/no questions, 14 had "yes" as the correct answer. Generators love writing questions that are true. A model that answered "yes" to everything would have scored 88 %. Psychologists have a name for this: acquiescence bias, the tendency to agree. Our test had it, not our model. The fix: the generator now alternates, so half the questions must be answered "no".
2. Two questions with the same name. Hand-written questions and generated ones ended up sharing IDs. The benchmark stored results by ID, so some questions were quietly scored against another question's sources. The benchmark now refuses to run if any ID appears twice.
3. Right answers marked wrong. The first scoring compared answer words literally. When a question was in English and the source in Swedish, a correct "February 15th" didn't match the expected "15 februari", and "0.5%" didn't match "0,5 %". Cross-language questions looked twice as bad as they were. The fix was a second language model as a judge, comparing meaning rather than spelling.
Who judges the judge?
A model grading another model sounds like letting students mark each other's homework. So I graded a sample myself and compared:
Agreement: the judge agreed with me on pass/fail for 17 of 22 answers. Letting it "think" before deciding raised that to 19 of 22.
Repeatability: at first, identical answers sometimes got different verdicts on different runs. Setting the judge's randomness to zero fixed that completely: 22 of 22 identical verdicts.
That looked solid. It wasn't the whole story; post 3 is about the day two well-behaved judges disagreed by 20 percentage points.
How many questions is enough?
Short answer: more than you'd like. A pass rate measured on n questions wobbles by roughly ±2·√(p(1−p)/n):
Questions per type | Margin at ~80 % pass rate |
|---|---|
15 | ±21 points |
30 | ±15 points |
100 | ±8 points |
400 | ±4 points |
So 15 questions per type gives a rough map: this type is weak, that one is fine. It can't prove a 5-point improvement. Two practical rules follow:
Compare question by question. Run the old and new system on the same questions and count which ones flipped from fail to pass and back. That's far more sensitive than comparing two averages.
Measure the noise first. I ran the identical setup twice: 23 of 273 results changed with nothing changed. Any improvement smaller than that is a coin toss in a lab coat.
What it was good for
With the battery in place, every later decision in this project became a measurement instead of an opinion:
It showed that search lost the right source in about 1 question in 5, and exactly where. That led to three fixes and a jump from 79 % to 84 %.
It caught a prompt rule that made the model begin a quarter of all answers with "Yes,".
It produced the bigger-model result from post 1.
It's portable: on a new customer's documents, the same tools generate a new battery in about an hour, without anyone knowing the answers in advance.
Takeaways
Write the answer key first. Generate questions from the documents and you know the answers without knowing the documents.
Check the generator. Literal answer words, no "the email", real paraphrases, real false premises. About 20 % won't survive, and that's fine.
Type your questions. A battery tells you what to fix; a single score only tells you how to feel.
Test the test. Look for yes-bias, duplicate IDs and scoring that can't read other languages.
Measure the noise, then compare question by question. Otherwise you're tuning to luck.
Next: "Even the yardstick needed calibrating". Two judges, one set of answers, 20 points apart, and what that means for every accuracy number you'll ever be shown.