Before blaming the model, log what you actually gave it. In most pipelines I review, the answer is a faithful summary of documents that had no business being in the context window.
The demo works. Ten carefully chosen questions, ten good answers, and everyone agrees it is ready. Two weeks after launch, support has a list of answers that are confidently, specifically wrong.
The instinct is to fix the prompt, then to try a better model. Both are usually wasted effort, because the model is not the component that failed. It was handed three passages, none of which contained the answer, and it did what it was asked to do with them.
Retrieval quality is a search problem, and search problems have numbers. This is the order I work in: build a small evaluation set, measure whether the right document is being retrieved at all, and fix the pipeline from chunking outward. The prompt comes last, and by then it usually needs one line.
Log the chunks, and most of the mystery disappears
The first change to make is not a fix, it is visibility. For every answer, store the query, the ids and scores of the retrieved chunks, the final prompt size, and the answer. Without that, every conversation about quality is anecdotal.
With it, triage takes minutes. Take ten complaints and look at what was retrieved. Nearly always they fall into one of three piles: the right document was never retrieved, the right document was retrieved but ranked below the noise, or the right document was retrieved and the model still went elsewhere. Only the third pile is a prompt or model problem, and it is usually the smallest.
This is also the moment to check something embarrassing but common: whether the document containing the answer is in the index at all. Ingestion pipelines fail quietly, and a superseded contract that was never re-indexed produces answers that look exactly like hallucinations.
| What the log shows | Diagnosis | Where to fix it |
|---|---|---|
| Correct chunk not in the results at all | Retrieval recall failure | Chunking, embeddings, or hybrid search — the biggest pile. |
| Correct chunk present but ranked 8th of 10 | Ranking failure | Re-ranking. Highest leverage for the least code. |
| Correct chunk ranked first, answer still wrong | Generation failure | Prompt, citations, or the model. The smallest pile. |
| Correct chunk exists but is outdated | Index freshness | Ingestion pipeline and deletion propagation. |
| Nothing relevant exists anywhere | The system should have abstained | Add a no-answer path. Silence beats invention. |
Build the eval set before changing anything
Fifty to a hundred real questions, each with the document or passage that genuinely contains the answer. Take them from support tickets and usage logs, not from imagination — the questions people actually ask are shorter, vaguer and more full of internal jargon than the ones you would write.
That set gives you two numbers. Retrieval recall at k: in what fraction of cases does the correct passage appear in the top k results? And answer faithfulness: how often is the final answer supported by the retrieved text? Separating them matters, because they have completely different fixes and improving one can hide a regression in the other.
Run it in CI. Once retrieval quality is a number in a pull request, the argument about whether a change helped stops being a matter of opinion — which is worth more over a year than any single improvement below.
Recall@k is the number to fix first. If the right passage is not in the context, no prompt, no model and no amount of temperature tuning will produce the right answer.
# cases.jsonl
# {"q": "what are the renewal terms for enterprise?", "doc_id": "msa-2026-v3#s4"}
# {"q": "can we cancel mid-term?", "doc_id": "msa-2026-v3#s7"}
def recall_at_k(cases, k=5):
hits = 0
for case in cases:
ids = [c.id for c in retrieve(case["q"], k=k)]
hits += case["doc_id"] in ids
return hits / len(cases)
def faithfulness(cases):
"""Is every claim in the answer supported by the retrieved context?
A cheap model as judge is fine here; what matters is that the number
moves consistently, not that it is perfectly calibrated."""
supported = 0
for case in cases:
ctx = retrieve(case["q"], k=5)
ans = generate(case["q"], ctx)
supported += judge_supported(ans, ctx)
return supported / len(cases)
# Print both on every pipeline change, and fail CI on a regression:
# recall@5 0.61 -> 0.94 faithfulness 0.78 -> 0.97Chunking is where most recall is lost
The default — split every document into fixed 512-token pieces — is the single largest cause of bad retrieval I see. It cuts tables in half, separates a clause from its heading, and strips the context that made a passage meaningful. A chunk reading “this shall not exceed 30 days” is unanswerable on its own and will never match a query about renewal notice periods.
Chunk on structure instead: headings, sections, list items, table rows. Keep the document title and section path inside the chunk text, not only in metadata, so the embedding itself carries the context. A chunk that begins “MSA 2026 › Section 4 › Renewal — …” retrieves far better than the same sentence alone, for the cost of a string concatenation.
Where chunks must be small for precision, use parent-document retrieval: index the small chunk, but hand the model the larger section it came from. You get the precision of small embeddings and the context of a whole section, which is usually the best available trade.
- Split on structure, never on a fixed token count alone: Headings, sections, list items, table rows. A fixed window that cuts through a table produces chunks that cannot answer anything.
- Put the title and section path in the chunk text: The embedding only knows what is in the string. Metadata that is not in the text does not help the vector find it.
- Retrieve small, pass large: Index precise chunks, give the model the surrounding section. Best precision, best context, one extra lookup.
- Keep tables and code intact: Serialise a table as one chunk, or as one chunk per row with the header repeated. Half a table retrieves as noise.
Vector search alone cannot find exact terms
Embeddings capture meaning, which is exactly wrong for the queries users actually type. Part numbers, error codes, customer names, contract ids, API endpoints, acronyms — these need lexical matching, and a vector search will happily return something semantically adjacent instead of the exact string.
Hybrid search fixes it: run a keyword query and a vector query, then fuse the results. Reciprocal rank fusion is the standard approach, it needs no score calibration between the two systems, and it is about fifteen lines of code. In the pipelines I have measured, hybrid alone typically takes recall@5 from around sixty percent into the eighties.
Most vector databases offer hybrid natively now, and Postgres does it well with `pgvector` alongside its own full-text search — which is worth knowing if you would rather not add another datastore for this.
def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
"""Fuse ranked id lists. No score normalisation needed, which is the
whole reason this is the default choice."""
scores: dict[str, float] = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
def retrieve(query: str, k: int = 50, tenant_id: str = "") -> list[str]:
dense = vector_search(query, k=k, filter={"tenant_id": tenant_id})
sparse = keyword_search(query, k=k, filter={"tenant_id": tenant_id})
return rrf([dense, sparse])[:k]
# The same idea directly in Postgres with pgvector + full-text search:
# WITH dense AS (
# SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS r
# FROM chunks WHERE tenant_id = $3 ORDER BY embedding <=> $1 LIMIT 50
# ), sparse AS (
# SELECT id, row_number() OVER (
# ORDER BY ts_rank_cd(tsv, plainto_tsquery($2)) DESC) AS r
# FROM chunks WHERE tenant_id = $3
# AND tsv @@ plainto_tsquery($2) LIMIT 50
# )
# SELECT id, sum(1.0 / (60 + r)) AS score
# FROM (SELECT * FROM dense UNION ALL SELECT * FROM sparse) u
# GROUP BY id ORDER BY score DESC LIMIT 50;Re-ranking is the highest return for the least code
Retrieval is optimised for recall; ranking is optimised for precision. Doing both with the same embedding model means doing neither well. The standard answer is to retrieve generously — fifty candidates — and then re-rank them with a cross-encoder that reads the query and each passage together and scores the pair directly.
It is more expensive per document than an embedding comparison, which is why you only run it on fifty candidates rather than the whole index. In exchange it is dramatically better at judging relevance, and it is the change that most often moves an unreliable pipeline into a reliable one.
Keep the top three to five after re-ranking, not the top twenty. Long contexts full of marginal passages actively hurt: they dilute attention, cost tokens, and give the model more opportunities to answer from the wrong place.
| Pipeline | Recall@5 | Unsupported answers |
|---|---|---|
| Fixed 512-token chunks, vector only | 61% | 22% |
| Structure-aware chunks, vector only | 74% | 14% |
| + hybrid keyword search (RRF) | 82% | 9% |
| + cross-encoder re-rank of top 50 | 94% | 3% |
| + abstention when top score is low | 94% | 1% (5% answered 'not found') |
Filters, freshness, and the failures that look like hallucination
Three operational problems produce answers indistinguishable from hallucination, and none of them is fixed by anything above.
The first is staleness. If your ingestion pipeline runs nightly and fails silently, the index confidently serves last month's pricing. Alert on ingestion lag and on documents whose source has changed but whose embedding has not. The second is deletion: when a document is removed or superseded, its chunks must go too, or the system will keep citing a contract that no longer exists.
The third is the one to be careful about. Tenant filtering must happen inside the retrieval query, not as a post-filter on the results, and never as an instruction in the prompt. Anything else risks one customer's documents informing another's answer — a breach that will be found eventually, and one a similarity threshold cannot excuse.
- Filter by tenant in the query, not afterwards: A post-filter throws away results you already retrieved, quietly reducing recall to near zero for small tenants — and one bug away from leaking.
- Alert on ingestion lag: Oldest un-indexed document age is the metric. A silent pipeline failure looks exactly like a model that has started lying.
- Propagate deletions and supersessions: Removing the source document must remove its chunks. Version documents so the retrieval can prefer the current one.
- Re-embed when you change embedding models: Query and document embeddings must come from the same model. A partial migration produces a silently broken index that still returns results.
Let the system say it does not know
Most RAG pipelines have no way to fail. Given no relevant context, the model is still asked to answer, so it does — from its own weights, in the same confident tone as everything else. Adding an explicit not-found path removes a whole category of wrong answers.
Two mechanisms, both cheap. Instruct the model to answer only from the provided context and to say when the answer is not there — and give it a concrete phrase to use, because models comply better with a template than with a principle. Then enforce it in code: if the top re-ranked score is below a threshold you have calibrated on the eval set, do not call the model at all. Return a not-found response with links to the closest documents.
Citations are the third piece, and they are worth the effort. Ask for the chunk id behind each claim, then verify programmatically that the ids exist in what you retrieved. A fabricated citation is an unambiguous signal you can log, alert on and show to the user — far more useful than a vague confidence score.
A pipeline that answers everything is not more useful than one that abstains sometimes — it is less useful, because nobody can tell which answers to trust.
ANSWER_PROMPT = """Answer only from the CONTEXT below.
Cite the chunk id for every claim, like [c7].
If the context does not contain the answer, reply exactly:
NOT_IN_CONTEXT
Do not use knowledge from outside the context."""
def answer(query, tenant_id):
candidates = retrieve(query, k=50, tenant_id=tenant_id)
ranked = rerank(query, candidates)[:5]
# Threshold calibrated on the eval set, not guessed
if not ranked or ranked[0].score < MIN_RELEVANCE:
return NotFound(closest=ranked[:3])
out = generate(ANSWER_PROMPT, query, ranked)
if "NOT_IN_CONTEXT" in out.text:
return NotFound(closest=ranked[:3])
# Verify every citation actually exists in what we retrieved
cited = set(extract_citations(out.text))
if not cited or not cited <= {c.id for c in ranked}:
log.warning("fabricated or missing citation", extra={"q": query})
return NotFound(closest=ranked[:3])
return Answer(text=out.text, sources=[c for c in ranked if c.id in cited])Keep measuring it after launch
Retrieval quality degrades on its own. Documents are added, the vocabulary of your customers shifts, a new product line arrives with terms nothing in the index has seen. A pipeline that scored ninety-four percent at launch will not stay there without attention.
Three things to keep running. The eval set in CI, so pipeline changes are checked before merge. A weekly sample of real queries with their retrieved chunks, reviewed by someone who knows the domain — an hour that consistently finds problems no automated metric catches. And a thumbs-down button whose feedback lands somewhere a human reads, with the retrieved chunk ids attached so the review takes seconds rather than an investigation.
Add new failures to the eval set as they are found. That is what turns the set from a one-off benchmark into a regression suite, and it is the difference between a pipeline that improves and one that merely changes.
Frequently asked questions
Why does my RAG system make things up on production data?
Usually it is not making things up — it is summarising the wrong retrieved passages faithfully. Log the chunks behind each answer and triage: either the correct passage was never retrieved, it was retrieved but ranked too low, or nothing relevant exists and the system had no way to abstain. Only the last category is really a model problem, and it is the smallest.
What chunk size should I use for RAG?
Chunk on structure rather than a fixed size — headings, sections, list items, table rows — and include the document title and section path inside the chunk text so the embedding carries that context. Where small chunks are needed for precision, index the small chunk but pass the model the larger parent section. Fixed 512-token windows that cut through tables and clauses are the most common cause of poor recall.
Do I need hybrid search, or is a vector database enough?
You need hybrid as soon as users type exact terms — part numbers, error codes, customer names, contract ids. Embeddings capture meaning and will return something semantically adjacent instead of the exact string. Combining keyword and vector results with reciprocal rank fusion is about fifteen lines and typically moves recall@5 from the low sixties into the eighties.
Is re-ranking worth the extra latency and cost?
In most pipelines it is the single highest-return change. Retrieve fifty candidates cheaply, then score query-passage pairs with a cross-encoder and keep the top three to five. It adds tens of milliseconds and a small cost, and it routinely takes recall@5 into the nineties — which then lets you send fewer tokens to the model, often paying for itself.
How do I stop RAG leaking data between tenants?
Filter by tenant inside the retrieval query itself, never as a post-filter on results and never as an instruction in the prompt. Post-filtering silently destroys recall for small tenants and is one bug away from leaking. The same applies to caching: any cached answer must be keyed by tenant and permission scope.
How many evaluation questions do I need?
Fifty to a hundred real questions with known correct sources is enough to steer every decision in the pipeline. Take them from support tickets and query logs rather than writing them, because real questions are shorter, vaguer and full of internal jargon. Then add every production failure to the set so it becomes a regression suite rather than a one-off benchmark.