Q5Agents & RAG

Q5. How do you test and monitor AI in production?

A practical setup for AI evaluation: a golden test set run on every change, shadow and canary rollouts, and production signals like time to first token, cost per session and how often users edit AI output.

Rajendra Kumar · · 2 min read

Q5. How do you test and monitor AI in production?

You can't eyeball AI output and hope for the best, and an HTTP 200 tells you nothing about whether the answer was right. I use three layers, and a loop that keeps feeding them.

Offline evals on every PR using a golden set of 100 to 200 cases, then shadow or canary rollouts comparing cost, latency and drift, then production monitoring of time to first token, cost per session and edit rate. A weekly failure review turns bad traces into new golden cases.
Architecture · Each layer catches what the one before it missed, and production failures flow back into the test set.

1. Before merge: the golden set

A versioned set of 100 to 200 real questions, heavy on the tricky edge cases. It runs automatically on every prompt, model or chunking change. I check retrieval (did the right document come back?) and the answer (is it faithful to the sources, relevant, and were the tool arguments correct?) using normal assertions plus an LLM judge where needed.

2. Before full rollout: shadow and canary

Run the new prompt or a cheaper model on live traffic in the background, without showing users, and compare cost, latency and output against the current version. Then move a small slice of real traffic over.

3. In production: the numbers I actually watch

  • Time to first token. How fast it feels.
  • Cost per session. Tokens spent per user, per task.
  • Edit and reject rate. How often people rewrite or reject what the AI produced. If users keep rewriting the drafts, something is broken, even if every other metric looks fine.

Key takeaways

  1. 01A golden set on every change, like unit tests.
  2. 02Shadow first, then canary, then everyone.
  3. 03User edit rate is the most honest quality metric.
AI evaluationEvalsObservabilityLLM-as-a-judgeInterview prep