
I treat LLM calls like any other cloud bill: look at what you actually use and pay only for what each task needs. Three levers do most of the work.

1. Route by difficulty
Not every task needs the most expensive model. A router (often just simple rules or a quick classifier) sends classification, tagging, summaries and extraction to fast, cheap models, and saves the frontier reasoning models for multi-step planning and hard synthesis.
2. Cache two ways
- Prompt caching. Put the static parts first: system instructions, tool definitions, long reference docs. Providers cache that prefix, which cuts input cost and time to first token.
- Semantic cache. Store answers to frequently repeated questions in Redis and return them without calling the model.
3. Stream, and push the rest to a queue
FastAPI streams tokens to the UI with Server-Sent Events, so the app feels fast even if the full answer takes a few seconds. Logging, eval scoring and memory extraction run in a background queue, off the user's request.
Key takeaways
- 01Most requests should never touch the biggest model.
- 02Static prompt prefix first, so it caches.
- 03Stream to the user; queue everything else.