
First I work out whether it's a search problem or a generation problem. Nine times out of ten it's search: the model answered faithfully from the wrong chunks.

Fixing search
- Hybrid search. Vector search is good at meaning but bad at exact matches like product codes, error codes or policy numbers. I combine it with keyword (BM25 / full-text) search, which in Postgres means pgvector plus built-in full-text search.
- Filter first. Narrow by tenant and category before the similarity search, not after.
- Better chunks. Replace fixed-size chunks with section-aware ones. Find small chunks for precision, then pass the parent section for context.
- Rerank. A cross-encoder reranker takes the top 20 candidates and keeps the best 5.
Fixing speed
Cache frequent questions in Redis, run the two searches in parallel, and stream the answer so the user sees the first words quickly.
Fixing made-up answers
The prompt requires a [source] citation for every claim. And if the best reranked score is below a threshold, the system says "I don't have enough information" instead of guessing. A clear refusal is much cheaper than a confident wrong answer about a refund policy.
This is the same hybrid setup I use in ResolveOps for answering from uploaded policy documents.
Key takeaways
- 01Check retrieval before you touch the prompt.
- 02Vector + keyword + reranker beats vector alone.
- 03Low confidence means "I don't know", not a guess.