This looks like a question we've answered before
Category: AI Engineering
Why serving a stored answer to a similar question went wrong 66 times out of 66, and the three checks that fixed it.
Posts 1 to 3 chose the model and built a test I could trust. This one is about the shortcut that looked too good to need measuring.
Four words this post leans on (the rest are in the pocket glossary in post 1):
Word | What it means here |
|---|---|
Vector (embedding) | A long list of numbers that describes what a text means. Two texts about the same thing get similar numbers, like two songs in the same genre. |
Similarity | How close two vectors are, from 0 (strangers) to 1 (twins). |
Threshold | The line I draw: above it, two questions count as the same. Picking it is the whole drama of this post. |
Semantic cache | A memory of answered questions, looked up by meaning instead of exact words. "Haven't we had this one before?" |
Every FAQ has a handful of questions that make up most of the traffic. Company A estimated that about 80 % of the questions their users ask are the same ones, over and over.
That suggests an obvious feature: let an admin approve the answer to a common question once, and serve that exact answer whenever someone asks the same thing again. It's instant (no language model needed), consistent (everyone gets the same vetted answer), and it improves as the admin approves more. Industry calls the general idea a semantic cache. I called it "approved answers", and in the plan it looked like the easiest win in the whole project.
It nearly became the most dangerous feature in the project.
The obvious design
The plan was standard. Turn every approved question into a vector (a numeric fingerprint of its meaning). When a new question arrives, turn it into a vector too and measure how similar the two are, from 0 (unrelated) to 1 (identical meaning). Above a threshold, say 0.85, serve the approved answer.
Before shipping it I wanted to know where to put the threshold, so I built a test.
The test
I took 40 questions from the test battery and treated them as "approved". For each one, a language model wrote four new questions:
Two paraphrases: the same question in other words. These should get the approved answer.
A different fact on the same topic: "When is the summer festival?" → "Where is the summer festival?" These must not.
One changed detail: "When is the summer festival 2026?" → "When is the summer festival 2025?" These must definitely not.
Every other approved question served as an unrelated control: 1,560 pairs that should never match.
The result
Here's how similar each group was to its approved question:
Group | Median similarity | Served at ≥ 0.85 | Served at ≥ 0.95 |
|---|---|---|---|
Paraphrase (should be served) | 0.970 | 80 / 80 | 63 / 80 |
Changed detail (must not) | 0.942 | 37 / 40 | 19 / 40 |
Different fact (must not) | 0.918 | 29 / 40 | 7 / 40 |
Unrelated (1,560 pairs) | 0.351 | 0 | 0 |
Read the first two rows again. To the embedding model, "When is the summer festival 2026?" and "When is the summer festival 2025?" are almost as alike as two honest rewordings of the same question. A year is one token in a sentence full of matching words. Meaning-vectors are great at "what is this about?" and terrible at "which one, exactly?"
At the textbook threshold of 0.85, all 66 near misses would have been served the approved answer. Every single one would have been a confident, wrong, admin-badged answer: the worst kind of wrong, because it carries a stamp that says "approved".
Raising the threshold doesn't rescue it. At 0.95, a fifth of the real paraphrases stop being served, and 26 near misses still get through. There is no number that separates the groups, because they overlap.
The good news hides in the last row: unrelated questions never come close. The danger is entirely in the "same topic, different specifics" zone. That happens to be exactly where FAQ users live: same report, different month; same setting, different page.
Attempt 2: a smarter similarity model
The system already had a reranker, a model that reads two texts together rather than comparing fingerprints. Surely that would notice "2025 ≠ 2026"?
It mostly didn't. Even combining a high similarity threshold with a strict reranker score still served 16 of 40 changed-detail questions. The reranker was trained to judge relevance: "is this text about the same thing?" A question about the 2025 festival is relevant to the 2026 festival. It just isn't the same question.
Attempt 3: rules
If the difference is in the details, check the details. I wrote a rule that extracts the specifics from both questions and requires them to match: numbers, dates (normalised, so "2016-06-06", "June 6, 2016" and "6 juni 2016" are the same), weekdays, months, seasons and words like "this year" or "last week".
Check | Paraphrases served | Near misses served |
|---|---|---|
Similarity only | 80 / 80 | 66 / 66 |
Similarity + detail rule | 79 / 80 | 34 / 66 |
Half the problem gone. But the other half is "different fact on the same topic": when vs where, postal code vs phone number. No date or number differs. I tried a rule for question words too ("who", "when", "where"), and it backfired spectacularly. It rejected 25 of 80 genuine paraphrases, because "When is the event…" and "What is the date of the event…" use different question words for the same question. Language, it turns out, is not a regex.
Attempt 4: ask a model
The remaining distinction needs understanding, so I asked a small language model directly: "Do these two questions ask for exactly the same information about the same thing? Reply YES or NO." Then I tested several models as that checker, on the 146 pairs that passed the similarity threshold:
Checker | Paraphrases served | Near misses served | Time per check |
|---|---|---|---|
gemma4:e4b (the answer model itself) | 80 / 80 | 14 / 66 | 0.26 s |
qwen3:4b-instruct | 80 / 80 | 9 / 66 | – |
qwen3:8b | 80 / 80 | 4 / 66 | 0.43 s |
qwen3:14b | 80 / 80 | 1 / 66 | 0.84 s |
Two side notes from this round:
A "thinking" variant of qwen3:4b answered NO to everything. Perfectly safe, perfectly useless: the AI equivalent of a doorman who keeps everyone out, including the owner.
I tried having two or three models vote, expecting the errors to cancel out. They didn't. The three voters (gemma4:e4b, qwen3:4b-instruct and qwen3:8b) all failed on the same four pairs. When the hard cases are hard for everyone, more voters just agree more confidently.
The best checker, qwen3:14b, didn't fit next to the answer model in the laptop's memory, so every check meant a seven-second model swap. Back to the trade-offs.
The combination that worked
Serve an approved answer only if all three checks agree:
Similar enough: a cheap first filter that throws out the 1,560 unrelated pairs.
Same details: the rule catches changed years, dates, numbers and seasons.
Same request: qwen3:8b, small enough to share memory with the answer model, catches "when" vs "where".
Paraphrases served | Near misses served | |
|---|---|---|
Similarity alone | 80 / 80 | 66 / 66 |
All three checks | 79 / 80 | 1 / 66 |
The rule and the model fail in different places, which is exactly why combining them works when voting between similar models didn't. The one remaining leak was "which year was the company founded" versus "which company was founded": no detail differs, and the model judged them the same. The approved answer most likely names both, but it's still the wrong question.
And the one paraphrase in 80 that isn't served? It simply gets a normally generated answer. A miss on this side costs a couple of seconds. A miss on the other side costs trust.
The safety rails around it
Measuring the matching was the big lesson, but a few more design choices keep approved answers honest:
Approved answers expire. Each one stores a fingerprint of the source texts it rests on. If a source changes or disappears, or a six-month review date passes, the answer stops being served until someone re-approves it.
The badge shows the matched question. Users see "✓ Approved answer to: When is the summer festival 2026?", so a human can spot a mismatch the checks missed.
There's always a way out. A "fresh answer from the sources" button generates a normal answer instead.
No "hint mode". The original plan included passing a nearly matching approved answer to the language model as a hint. Dropped: after this test, handing the model an answer to a different question looked like a recipe for exactly the error we'd just fought off.
Live, served approved answers came back in 0.4–4 seconds, against 5–7 for a generated answer. "Which number does kayak X have?" got the approved answer. "Which number does kayak Y have?" didn't, and the model answered it correctly from the sources.
Takeaways
Similarity isn't sameness. Embeddings are built to find what's about the same thing. FAQ questions differ in the specifics, which embeddings barely register.
Test the cache before you trust it. Paraphrases, different facts, changed details and unrelated controls. If you don't build the near misses yourself, your users will find them for you.
Unrelated questions aren't the risk; near misses are. Any threshold handles the easy cases.
Combine checks that fail differently. A rule and a model beat several models voting, because similar models make the same mistakes.
Make the failure cheap and visible. When unsure, generate instead of serving; show which question was matched; let answers expire when their sources change.
Or, in one line: a semantic cache is a great way to give the right answer to the wrong question, very quickly.
Next: swapping the database. Why the switch from Qdrant to Postgres with pgvector first made search clearly worse, and the five fixes that brought it back.