A chat feature returns text. An agent changes something in the world — and everything difficult about this architecture follows from that one difference.
Agentic features have a specific failure signature. They work beautifully in a notebook, survive a demo, and then fall over in production the first time a real workflow takes ninety seconds, calls four tools and hits a rate limit on the third.
The cause is almost never the model. It is that a multi-step, minutes-long, side-effect-producing process has been wired into an HTTP request, which has a timeout, no memory, and no way to resume. When it fails at step four, the three expensive steps before it are lost and the user's only option is to press the button again — which runs step one to three all over again, and may well send the same email twice.
None of this is new. Long-running work with side effects has been a solved problem in backend engineering for decades; what is new is that each step costs real money and takes seconds rather than milliseconds, which makes the discipline matter more, not less.
The thirty-second wall, and why retrying makes it worse
Every layer between your user and your code has a timeout, and they are shorter than agent runs. Load balancers commonly default to sixty seconds, API gateways to thirty, browsers and mobile clients to whatever the user's patience allows. The request is abandoned; the work carries on invisibly, or dies halfway through.
The damage is not the timeout itself, it is what surrounds it. The client retries, so the workflow starts again from the beginning. The user closes the tab, and nobody knows whether the CRM was updated. A failure in the last step discards everything spent on the first four, and with model calls that is money rather than just time.
Streaming tokens back is not a fix. It keeps the connection warm while the model talks, but an agent spends most of its run calling tools, not producing text, and the connection has nothing to stream during those windows.
| Failure | What the user sees | What it costs you |
|---|---|---|
| Gateway timeout at 30s | An error, on work that is still running | The full run, billed, with no result |
| Client retry after timeout | Sometimes duplicate results | The run again — and possibly a duplicate side effect |
| Rate limit on step 3 of 5 | Generic failure | Steps 1 and 2 discarded and re-run on the next attempt |
| Tab closed mid-run | Nothing, ever | A completed run nobody collects |
| Deploy during a run | Silent disappearance | Every in-flight run in that process |
The shape: accept, enqueue, return an id
The whole architecture follows from one decision: the HTTP endpoint does not do the work. It validates the request, records a run, enqueues it and returns 202 with a run id. Everything else happens in a worker, and the client follows progress separately.
This is worth the small amount of extra frontend work several times over. The run survives deploys, tab closes and network drops. Its progress is queryable by anyone with the id, so support can answer “what happened to my report” without reading logs. And because the run has a row, you can attribute cost, latency and failure to it later.
For progress, polling a status endpoint every second or two is perfectly good and by far the simplest thing that works. Reach for server-sent events when you want token-by-token output during the generation steps — but keep the run's authoritative state in the database, not in the stream, so a dropped connection loses nothing.
Checkpoint after every step, before starting the next. That one habit turns a failed run from a full replay into a resume, and with model calls the difference is measured in money.
// POST /api/agent-runs -> 202 { runId }
app.post('/api/agent-runs', async (req, res) => {
const input = AgentInput.parse(req.body)
await enforceQuota(req.user.tenantId)
const run = await db.agentRuns.create({
tenantId: req.user.tenantId,
userId: req.user.id,
input,
status: 'queued',
steps: [],
})
await queue.add('agent.run', { runId: run.id }, {
attempts: 1, // the WORKFLOW does not retry; steps do
removeOnComplete: false, // keep the record for cost attribution
})
res.status(202).json({ runId: run.id })
})
// GET /api/agent-runs/:id -> { status, steps, result, costUsd }
// The client polls this, or subscribes for token streaming while the
// authoritative state stays in the row.
// The worker: each step is checkpointed before the next begins
worker.process('agent.run', async (job) => {
const run = await db.agentRuns.find(job.data.runId)
for (const step of plan(run.input)) {
if (run.completedSteps.includes(step.name)) continue // resume, not restart
const result = await runStep(step, run) // own retry + timeout
await db.agentRuns.appendStep(run.id, step.name, result)
if (result.status === 'needs_approval') {
return db.agentRuns.setStatus(run.id, 'awaiting_approval')
}
}
await db.agentRuns.setStatus(run.id, 'complete')
})Steps, not one big job
Treat each tool call and each model call as its own step, with its own timeout, its own retry policy and its own recorded result. A run is then a sequence of durable facts rather than one opaque function that either worked or did not.
The payoff is concentrated in failure. A rate limit on the search step retries the search, not the plan and the draft that preceded it. A validation failure on the CRM write can be retried after a fix without regenerating anything. And when someone asks why a run cost eighty cents, the answer is a table rather than an investigation.
Steps also give you sane timeouts. A model call might warrant sixty seconds; a CRM write should fail in five. One timeout for the whole run cannot express that, so it ends up set to the longest thing that might happen — which means a hung HTTP call holds a worker for minutes.
- One row per step, written before moving on: Name, input hash, output, status, duration, tokens, cost. It is the difference between debugging and guessing.
- Retry policies belong to the step, not the run: Retry a rate-limited model call with backoff; do not retry a validation failure at all; retry a flaky vendor twice and then stop.
- Cap the run, not just the step: A maximum number of steps and a maximum spend per run. Agents loop, and the loop is billed.
- Version the plan: Store which version of the workflow a run used, so a deploy mid-run does not resume into a different set of steps.
Idempotency, because at-least-once is what you actually get
Queues deliver at least once. Workers are killed mid-run by deploys and scaling events. Retries happen after a step has already succeeded but before its result was recorded. Assume every side effect can be attempted twice, because eventually each one will be.
The fix is an idempotency key derived from the run and the step — not a random value generated at call time, which defeats the purpose. Pass it to any API that supports one; for those that do not, keep a small table of performed effects and check it before acting.
This matters far more for agents than for ordinary background jobs, because their side effects are outward-facing. A duplicate thumbnail is invisible. A duplicate email to a customer, a duplicate CRM contact, a duplicate refund — those are the incidents that make a company turn an AI feature off.
// Deterministic: same run, same step, same key. Never randomUUID() here.
const idempotencyKey = `${run.id}:${step.name}:${hash(step.input)}`
// 1. APIs that support it (payments, most modern SaaS APIs)
await crm.contacts.create(payload, { idempotencyKey })
// 2. APIs that do not: guard with your own table
async function once<T>(key: string, fn: () => Promise<T>): Promise<T> {
const existing = await db.effects.find(key)
if (existing) return existing.result as T // already done, return it
// Unique constraint on key: two workers racing, one wins
const claim = await db.effects.claim(key)
if (!claim) return (await db.effects.waitFor(key)).result as T
const result = await fn()
await db.effects.complete(key, result)
return result
}
await once(idempotencyKey, () => mailer.send(draft))
// 3. And make the model's tool calls idempotent by construction where
// you can: "upsert contact by email" beats "create contact".Retries that do not multiply the bill
Retrying an agent step is not like retrying a database query. Each attempt costs tokens, and a naive policy applied to a five-step workflow can quietly triple the cost of every failure.
Classify errors before retrying. Rate limits and transient server errors deserve exponential backoff with jitter and a small attempt cap. Timeouts deserve one retry and then a hard stop, since something is usually wrong upstream. Validation failures and refusals should never be retried unchanged — feed the error back once as context, then give up and degrade.
Give every run a spend ceiling and enforce it between steps. It is the only reliable defence against the agent that loops, calls the same tool eleven times and produces a four-dollar invoice for a task that should cost six cents. Log the cap being hit as a product signal, not just an error: it usually means the plan is wrong, not that the budget is too small.
| Failure | Retry | Why |
|---|---|---|
| 429 rate limit | Yes — backoff with jitter, up to 3 | Transient and expected under load; jitter prevents a thundering herd. |
| 5xx from the provider | Yes — 2 attempts | Usually transient. Beyond two, degrade rather than keep paying. |
| Request timeout | Once | If it timed out twice, the problem is not luck. |
| Schema validation failure | Once, with the error fed back | A second attempt with feedback often succeeds; a third rarely does. |
| Model refusal or policy block | No | It will refuse again. Route to a fallback or return a clear message. |
| Tool returned a business error | No | Not a transient fault. Surface it to the user or the approval queue. |
Queue or workflow engine?
Both answers are legitimate and the choice is mostly about how much state the process carries. A queue plus your own run table is simpler, has fewer moving parts, and is the right call for workflows of a handful of steps that complete in minutes.
A durable workflow engine — Temporal, or a cloud state machine service — earns its complexity when runs wait for hours or days, when human approval sits in the middle, when there are many steps with complex branching, or when you need versioning so in-flight runs survive deploys of changed logic. It gives you determinism, replay and visibility you would otherwise be rebuilding by hand.
The wrong move is adopting the heavier tool before you have felt the problem it solves. Start with a queue, keep the run state in your own database, and let the shape of the failures tell you when you have outgrown it. Most products never do.
| Signal | Queue + run table | Durable workflow engine |
|---|---|---|
| 3–8 steps, finishes in minutes | Yes | Overkill |
| Waits hours or days for a human | Awkward | Yes — this is what it is for |
| Complex branching and compensation | Gets messy fast | Yes |
| Deploys must not break in-flight runs | You will build versioning yourself | Built in |
| Small team, no platform engineer | Yes | Consider the operational cost honestly |
| Already running Redis or SQS | Yes — use what you have | Only if the above signals apply |
Humans in the loop, and permissions on tools
The fastest way to make an acting agent acceptable to customers is to stop it before the consequential step. An approval gate is a run state, not a UI feature: the worker records what it intends to do, sets the run to awaiting approval, and exits. A human reviews the proposed action and approves or rejects; approval enqueues the remaining steps.
Give approvals a timeout and a default. A run that waits forever is a leak — of context, of relevance, sometimes of a lock. Expire after a day, notify, and require a fresh run.
Then scope what tools can do. The agent should execute with the permissions of the user who started the run, never with a service account that can do everything. Write operations should be narrow and specific — `create_draft_email` rather than `send_email`, `propose_refund` rather than `issue_refund` — because a tool that can only propose cannot cause an incident, and an audit trail of what was proposed, approved and executed is what makes the feature defensible when someone asks.
- Approval is a state, not a modal: Persist the proposed action, exit the worker, resume on approval. A pending approval must survive a deploy and a restart.
- Run with the user's permissions: An agent holding a service account that can do anything is a privilege escalation waiting for a prompt injection to find it.
- Prefer proposing to doing: Draft, propose, stage. Reversibility is worth more than autonomy in almost every B2B workflow.
- Log every tool call in an audit trail: Who started the run, what the model proposed, who approved, what executed. This is the artefact that answers the security questionnaire.
Watch runs, not requests
Standard web monitoring tells you nothing useful here. The endpoint returns in forty milliseconds and always succeeds; the interesting behaviour is in the runs, and they need their own dashboard.
Five numbers cover it: completion rate, p95 duration, cost per run, the distribution of which step fails, and the count of runs stuck in a non-terminal state for too long. That last one is the alert that matters most, because a stuck run is invisible to everything else — no error, no exception, just a customer wondering where their report is.
Dead-letter anything that exhausts its retries, with the full step history attached, and review the queue weekly. Failed agent runs are the best product feedback available: they show exactly where the plan, the tools or the prompts do not match what users are actually asking for.
-- Which step fails, and what each failure costs
SELECT
failed_step,
count(*) AS runs,
round(avg(cost_usd)::numeric, 3) AS avg_cost,
round(avg(duration_ms) / 1000.0, 1) AS avg_seconds
FROM agent_runs
WHERE status = 'failed' AND created_at > now() - interval '7 days'
GROUP BY failed_step
ORDER BY runs DESC;
-- Stuck runs: the alert nothing else will give you
SELECT id, tenant_id, status, now() - updated_at AS stalled_for
FROM agent_runs
WHERE status NOT IN ('complete', 'failed', 'cancelled')
AND updated_at < now() - interval '15 minutes'
ORDER BY updated_at;Frequently asked questions
Why does my AI agent time out in production but work locally?
Locally nothing enforces a deadline. In production the load balancer, API gateway and client each impose one — commonly thirty to sixty seconds — and an agent making several tool and model calls exceeds that routinely. The fix is not a longer timeout; it is to stop doing the work inside the request. Accept, enqueue, return a run id, and report progress separately.
Do I need Temporal, or is a normal queue enough?
A queue plus your own run table is enough for workflows of a handful of steps that finish in minutes, and it is far less to operate. A durable workflow engine earns its complexity when runs wait hours for human approval, when branching and compensation get complex, or when in-flight runs must survive deploys of changed logic. Start with the queue and let the failures tell you when you have outgrown it.
How do I stop an agent doing the same thing twice?
Derive an idempotency key from the run id, the step name and a hash of the step input — never a random value — and pass it to APIs that support one. For APIs that do not, keep a table of performed effects with a unique constraint on the key and check it before acting. Queues deliver at least once, so duplicate attempts are certain over time, not hypothetical.
How should I show progress to the user?
Poll a status endpoint every second or two. It is simple, survives dropped connections, and works across tabs and devices. Add server-sent events when you want token-by-token output during generation, but keep the authoritative run state in the database so nothing is lost when the stream drops.
How do I keep agent runs from getting expensive?
Cap steps per run and spend per run, and enforce both between steps rather than only at the start. Classify failures before retrying — never retry a refusal or a validation error unchanged — and checkpoint after every step so a failure resumes rather than replays. Then track cost per run as a first-class metric next to completion rate.
Should an AI agent be allowed to take actions automatically?
For reversible, low-consequence actions, yes. For anything a customer sees or money touches, put an approval gate in front and give the agent tools that propose rather than execute — create a draft, stage a change, propose a refund. Run with the initiating user's permissions rather than a service account, and keep an audit trail of proposed, approved and executed actions.