
You can't eyeball AI output and hope for the best, and an HTTP 200 tells you nothing about whether the answer was right. I use three layers, and a loop that keeps feeding them.

1. Before merge: the golden set
A versioned set of 100 to 200 real questions, heavy on the tricky edge cases. It runs automatically on every prompt, model or chunking change. I check retrieval (did the right document come back?) and the answer (is it faithful to the sources, relevant, and were the tool arguments correct?) using normal assertions plus an LLM judge where needed.
2. Before full rollout: shadow and canary
Run the new prompt or a cheaper model on live traffic in the background, without showing users, and compare cost, latency and output against the current version. Then move a small slice of real traffic over.
3. In production: the numbers I actually watch
- Time to first token. How fast it feels.
- Cost per session. Tokens spent per user, per task.
- Edit and reject rate. How often people rewrite or reject what the AI produced. If users keep rewriting the drafts, something is broken, even if every other metric looks fine.
Key takeaways
- 01A golden set on every change, like unit tests.
- 02Shadow first, then canary, then everyone.
- 03User edit rate is the most honest quality metric.