Q7Architecture

Q7. How do you keep LLM cost and latency low as traffic grows?

Three levers for LLM unit economics in a SaaS product: route easy tasks to small models, cache with prompt caching and Redis, and stream responses from FastAPI while pushing post-processing to a queue.

Rajendra Kumar · · 2 min read

Q7. How do you keep LLM cost and latency low as traffic grows?

I treat LLM calls like any other cloud bill: look at what you actually use and pay only for what each task needs. Three levers do most of the work.

A request checks a Redis semantic cache, then a router sends most tasks to a small model and a few to a frontier model. Both stream through SSE to the UI. Prompt caching keeps static prefixes first, and logs and evals go to a background queue.
Architecture · Cheap model by default, cache what repeats, stream everything, and keep the slow stuff off the request path.

1. Route by difficulty

Not every task needs the most expensive model. A router (often just simple rules or a quick classifier) sends classification, tagging, summaries and extraction to fast, cheap models, and saves the frontier reasoning models for multi-step planning and hard synthesis.

2. Cache two ways

  • Prompt caching. Put the static parts first: system instructions, tool definitions, long reference docs. Providers cache that prefix, which cuts input cost and time to first token.
  • Semantic cache. Store answers to frequently repeated questions in Redis and return them without calling the model.

3. Stream, and push the rest to a queue

FastAPI streams tokens to the UI with Server-Sent Events, so the app feels fast even if the full answer takes a few seconds. Logging, eval scoring and memory extraction run in a background queue, off the user's request.

Key takeaways

  1. 01Most requests should never touch the biggest model.
  2. 02Static prompt prefix first, so it caches.
  3. 03Stream to the user; queue everything else.
LLM costModel routingPrompt cachingStreamingFastAPIInterview prep