Q4Agents & RAG

Q4. Your RAG system is slow and makes things up. How do you fix it?

How I fix a RAG pipeline that is slow and hallucinates: hybrid vector and keyword search, tenant pre-filtering, reranking, Redis caching, citations and an abstain threshold.

Rajendra Kumar · · 2 min read

Q4. Your RAG system is slow and makes things up. How do you fix it?

First I work out whether it's a search problem or a generation problem. Nine times out of ten it's search: the model answered faithfully from the wrong chunks.

Query checks a Redis cache, then pre-filters by tenant and category, runs vector search and BM25 keyword search in parallel, reranks the top 20 down to 5, and either sends them to the LLM with citations or, if scores are low, answers that there isn't enough information.
Architecture · Hybrid search plus a reranker fixes most bad answers. The threshold catches the rest.

Fixing search

  • Hybrid search. Vector search is good at meaning but bad at exact matches like product codes, error codes or policy numbers. I combine it with keyword (BM25 / full-text) search, which in Postgres means pgvector plus built-in full-text search.
  • Filter first. Narrow by tenant and category before the similarity search, not after.
  • Better chunks. Replace fixed-size chunks with section-aware ones. Find small chunks for precision, then pass the parent section for context.
  • Rerank. A cross-encoder reranker takes the top 20 candidates and keeps the best 5.

Fixing speed

Cache frequent questions in Redis, run the two searches in parallel, and stream the answer so the user sees the first words quickly.

Fixing made-up answers

The prompt requires a [source] citation for every claim. And if the best reranked score is below a threshold, the system says "I don't have enough information" instead of guessing. A clear refusal is much cheaper than a confident wrong answer about a refund policy.

This is the same hybrid setup I use in ResolveOps for answering from uploaded policy documents.

Key takeaways

  1. 01Check retrieval before you touch the prompt.
  2. 02Vector + keyword + reranker beats vector alone.
  3. 03Low confidence means "I don't know", not a guess.
RAGpgvectorHybrid searchRerankingRedisInterview prep