Category: AI Engineering/LLM evaluation

How I chose, benchmarked and re-chose the language model for a local AI FAQ, and why the 15-billion-parameter model got outread by one that does its thinking with 4.

Here is the result first

because it surprised me

On the same questions, with the same retrieved sources and graded by the same judge, a small 2026 model beat a model with almost twice its parameters, and nearly four times its working parameters:

The bigger model was four times slower, made three times as many wrong answers, and passed fewer questions.

Size, it turns out, isn't everything. (Your mother could have told you that, but she doesn't do benchmarks.)

This post is about how I got there: what I chose and why, how I measured, what made me change my mind twice, and what I'd tell anyone choosing a model for a retrieval-based assistant.

The job

Company A, a Swedish software company, wanted a "smart FAQ": an assistant that answers their users' questions from their own documentation. Their constraints shaped everything that followed:

• Nothing leaves their servers. No cloud LLM APIs: the model, the data and the logs all run in-house.

• The model must be replaceable. We called it the "empty box": everything the system learns has to survive a switch to next year's model. No fine-tuning, no knowledge baked into weights.

• It must be fast. Their words: "if people wait 10 seconds, they leave".

• The production hardware is one RTX 5090 with 32 GB of memory.

The technique is RAG (retrieval-augmented generation). For each question the system finds the most relevant passages in the documentation, hands them to a language model, and the model writes an answer that cites them. The model doesn't need to know anything; it needs to read well and admit when the sources don't answer the question.

Testing on data I could judge

Before Company A's documentation was available, I needed a test bed where I could tell a right answer from a wrong one. I used my own email, chat and calendar archive. After cleaning, 10 GB of raw mail became about 34,000 short text sections. When the assistant says something about my own life, I know whether it's true. At least sometimes :)

The development machine was a MacBook (M4 Pro, 24 GB), which put a hard ceiling on model size. The stack was open source throughout:

• Ollama to run models locally and swap them with one setting.

• BGE-M3 to turn text into vectors (strong on Swedish).

• Qdrant as the vector database. Later this became Postgres with pgvector, because Company A already runs Postgres.

• Search by meaning and by keywords together, then a second small model (a reranker) to pick the best passages.

Every part sits behind a small interface, so the language model is just one replaceable component.

Round 1: pick the obvious winner

The first benchmark was modest: a handful of questions with known answers, plus "trap" questions whose answer is not in the archive and must be refused.

gemma3:4b won on both speed and accuracy. The losers were instructive:

• One model invented an answer to a trap question.

• One "thinking" variant leaked its reasoning into the answer text.

• The 30B-class models didn't fit in 24 GB at all.

Even in this first round, the 14B model was the slowest and no more accurate. I noted it and moved on.

The real lever wasn't the model

The early failures were mostly not about model quality:

• Small models are bad at dates. Asked when something happened, they quoted "over the weekend" straight from an email. Asked for a weekday, they guessed.

• They answered in the wrong language, writing Swedish to English questions.

• Old information beat new. Asked what someone does today, the model used a five-year-old email.

The fixes went into the system, not the model. The code now computes facts and hands them over: "Tuesday 2018-11-13, 7 years and 10 months ago". The prompt names the answer language. Questions about "now" also search the most recent sources and mark the newest one. On the weaker of the two models tested, answer accuracy went from 0.52 to 0.80 without touching the model.

That became a rule: improve the system so every model benefits, and keep model quirks in a small per-model profile. Swapping a model stays a one-line change.

Round 2: where does the time go?

The system worked, but answers took 7–9 seconds. Before trying faster models I measured where the time went:

• Reading the sources took about 75 % of the answer time. About 4,400 tokens of input against 85 of output.

• Fewer sources made answers faster but worse (accuracy 0.83 → 0.63).

• Trimming each source to its most relevant passage kept the quality and cut the time by about a quarter.

Then a second model round on 29 questions:

Again the bigger models were slower and no better. What changed the picture was the runtime, not the model. The same gemma4:e4b running on Apple's MLX engine, together with source trimming, came down to 4.0 seconds, from 8.9 for the old default. It was also the only one of the two finalists that refused a trap question (1 of 2; the other refused none). It became the new default.

Building a real yardstick

Twenty-nine questions can't show small differences. I needed a proper test, and I borrowed the idea from psychometrics: an intelligence test is a battery of subtests, each measuring one ability.

• The battery: 273 questions in typed groups. Lookups, paraphrases, names, numbers, yes/no, questions in another language than the source, typos and keyword-style queries, plus questions that must be refused: off-topic, general knowledge, leading questions and false premises.

• Questions with known answers, without knowing the archive. A model reads a random passage and writes a question from it, so the right source and the right answer are both known. Automatic checks throw away bad questions, about one in five.

• A second model grades the answers, because keyword matching fails across languages and formats ("February 15th" vs "15 februari").

Then the most important step: I ran the identical setup twice. 23 of 273 results changed with nothing changed. Language models and judges both vary a little. From then on, a change only counted if it beat that noise, and every comparison was made question by question (how many fixed, how many broken), never as two averages.

Round 3: surely a bigger model reads better?

By now the system found the right source most of the time. The remaining errors happened with the right source in hand: the small model misread it. The textbook answer is a bigger model, so I tested the biggest one that fit on the laptop.

The setup was as fair as I could make it:

• Same sources: one benchmark run, both models answering from identical search results, so only the model differed.

• Same judge, a third model that wasn't one of the two being tested. It agreed with my own hand ratings on 20 of 22 calibration answers.

Question by question, the bigger model fixed 16 answers and broke 27. That's more than the noise between two identical runs, and it points the same way on every measure. On the 114 original, non-variant questions it passed 87 against 97.

Same suitcase, packed differently

The first suspicion when a big model underperforms is compression. Models are quantized to fit in memory: their numbers are stored in 4 bits instead of 16, which loses some precision. Maybe the big one was simply squeezed harder?

It wasn't. I checked what actually ran:

Both were squeezed to about 4 bits, and they took up the same 9.4-ish GB of memory. "The bigger model" wasn't heavier at all. It was more parameters packed tighter into the same suitcase. In boxing terms, this was a same-weight-class fight, and the heavyweight's reputation didn't help.

So why did the smaller one win? I didn't isolate the cause, so these are hypotheses, not findings:

• It's a year newer. Small models improve fast, and what this job needs isn't encyclopedic knowledge but disciplined reading: stick to the sources, cite them, say "I don't know". Newer training seems to do exactly that better.

• It's built to be compressed. Google publishes Gemma 4 E4B in quantization-aware trained (QAT) versions: the model is trained knowing it will be squeezed to 4 bits, so it loses less when it is. Qwen3-14B at Q4_K_M is compressed after the fact. (Whether my particular MLX build came from the QAT weights, I couldn't confirm.)

• It's built for small, focused jobs. E4B is designed for on-device use. Whatever general knowledge it traded away to get there, a RAG system doesn't need. The answers have to come from the sources anyway. Paying for a model's extra knowledge here is like hiring a professor to read the manual aloud.

What the model catalogues can and can't tell you

I looked for a shortcut: a Hugging Face category for "good at answering from supplied sources". There isn't one. Both models are tagged as general text generation (Gemma even as multimodal "any-to-any"), which says nothing about whether a model sticks to the text it was given. The closest public yardsticks I found:

• the Vectara Hallucination Leaderboard, which measures how often a model adds facts that aren't in the text it summarises. That's close to what RAG needs.

• EuroEval, which ranks models in European languages, Swedish included.

• a handful of new RAG faithfulness datasets, none in Swedish yet.

They're useful for shortlisting candidates. But none of them knows your documents, your language mix or your users' odd phrasings, so the final choice still has to come from your own test.

The yardstick needed calibrating too

One more finding made me careful about all of the above. I graded the same 249 answers with two different judge models. Both had looked trustworthy on my 22 hand-rated calibration answers. One passed 55 %, the other 79 %. When I settled a sample of their disagreements by hand, each judge was right half the time: one was too strict, the other passed refusals and wrong numbers.

So absolute accuracy figures carry roughly ±12 points of judge uncertainty. The model comparison above still holds, because both models were graded by the same judge, and the judge-free keyword measure agrees. But I stopped quoting single accuracy numbers without a range.

Other "obvious" improvements that lost

The bigger model wasn't the only intuition that failed the test:

• A verification step, where a second model checks every claim against the sources before the answer is shown, raised precision by about 3 points but cost 8–15 seconds per answer on the laptop. It's off for now, to be re-measured on the GPU.

• Answer caching by similarity ("this question looks like an approved one, serve the approved answer") would have served the wrong approved answer to 66 of 66 "same question, different year or person" variants. The two kinds of question were almost equally similar: 0.94 vs 0.97. The fix was three checks instead of one, which brought it to 1 of 66.

• A general prompt rule for yes/no questions made the model start a quarter of all answers with "Yes,". The rule now only fires for real yes/no questions.

The verdict

• gemma4:e4b stays. It's the most accurate, by far the fastest, and refuses what it should. On the client's RTX 5090 it will be faster still.

• Newer beats bigger, at least here. For a model whose job is to read supplied sources carefully, model generation mattered more than parameter count.

• The choice isn't final, and that's the point. Because nothing is trained into the model, the next model is a one-line switch, then a run of the same 273 questions and a question-by-question comparison. A larger new model on the 32 GB card is the next candidate. It gets the same test, and it has to win on numbers, not on size.

What I'd tell anyone choosing a model for RAG

1. Fix the system before you blame the model. Computed dates, the right language and fresh sources did more than any model swap.

2. Measure where the time goes. Reading the sources dominated. The runtime engine and trimming mattered more than model size.

3. Measure the noise before the effect. Run the same test twice. If a change flips fewer results than that, it hasn't done anything yet.

4. Compare question by question, on identical sources, with one judge. Two averages hide everything.

5. Calibrate the judge, and report accuracy as a range.

6. Don't assume bigger is better. Test it. In this project it was slower, wrong more often, and passed fewer questions.

Next in the series: how I built a 273-question test battery without knowing the answers, and why "psychometrics for AI" turned out to be more than a metaphor.