A human developer who hits a rate limit reads the error, waits a bit, and tries again — there's a natural pause built into the fact that a person has to notice and act. An agent has no such pause by default. The instant a tool call returns a 429 or a timeout, the obvious next step is "call it again," and an agent can do that in milliseconds, in a loop, and — if it has spawned parallel sub-agents — from several places at once. A single struggling dependency turns into a self-inflicted denial-of-service, aimed by the agent at the exact service it needed to succeed. Backoff, jitter, and circuit breakers exist to put the pause back in, deliberately, before the agent's own retry behavior becomes the outage.
Cost budgeting asks "how much are we willing to spend before stopping" — a ceiling on total spend or token count (see cost and token budgets for autonomous agents). Idempotent tool design asks "is it safe to retry this specific call" — whether a repeated request produces the same effect as one (see designing tools for AI agents to call). Rate-limit and backoff design asks a third, different question: "given that a retry is safe and affordable, how fast should it happen, and when should we stop trying altogether." A system can have a generous budget and perfectly idempotent tools and still take down a dependency, because nothing in either of those properties slows the retry rate down. All three controls are independent and all three are needed — this guide is about the one that's usually missing.
| Signal | What it means | Right response |
|---|---|---|
| HTTP 429 / Retry-After header present | The server is explicitly telling you to slow down and for how long | Wait at least the stated duration, then retry — ignoring Retry-After to retry sooner is the single most common cause of repeated 429s |
| HTTP 5xx / connection timeout | Transient server-side trouble, possibly load-related | Exponential backoff with jitter, capped attempts |
| HTTP 400 / 422 (malformed request) | The request itself is wrong; retrying it unchanged will fail identically forever | Do not retry — fix the request or surface the error, since retrying just adds load for zero chance of success |
| HTTP 401 / 403 | Auth or permission problem, not a transient one | Do not retry the same call — escalate, since retries will not fix an expired token or a missing scope |
Treating every non-2xx response as "try again" is the most common version of this bug: it doesn't distinguish a server asking for patience from a request that's permanently broken, and it retries both the same way — burning attempts on the second category that could never have succeeded, while under-responding to the first.
Plain exponential backoff — wait 1s, then 2s, then 4s, then 8s — sounds sufficient, but if a dependency briefly fails for everyone at once (a deploy, a blip), every caller following the identical schedule retries at the identical moment, producing a new synchronized spike exactly when the dependency is trying to recover. Adding randomized jitter to each wait — say, a random value between 50% and 100% of the computed backoff — spreads those retries out instead of clustering them, which is the difference between a dependency recovering smoothly and a dependency getting hit by a second thundering herd seconds after the first one subsides. This matters more, not less, for agents than for humans, since an agent's retries are fast and mechanically identical in a way a human's never are.
Backoff slows retries down; it doesn't stop them. If a dependency has failed the last N consecutive calls, continuing to retry it — even slowly — still adds load to something that's demonstrably not recovering, and still burns the run's time budget on calls that are very unlikely to succeed. A circuit breaker tracks recent failure rate and, past a threshold, "opens": further calls fail fast without hitting the network at all, for a cooldown period, after which a single trial call decides whether to close the circuit again. For an agent this is the difference between "tool unavailable, stopping this approach" surfacing in seconds versus the agent silently retrying the same failing call for the rest of its turn budget while the actual task goes nowhere.
A single agent respecting backoff is not enough if it has spawned five sub-agents that each independently call the same rate-limited API — from the API's point of view, five well-behaved individual backoff schedules can still add up to the same aggregate hammering as one badly-behaved one. The fix is a shared state: a token bucket, a circuit-breaker flag, or a rate counter that all callers against the same downstream dependency read and write, so one sub-agent's 429 slows down its siblings too, not just itself. Without shared state, per-agent backoff optimizes each caller's individual politeness while leaving the aggregate behavior exactly as aggressive as no backoff at all.
The test to apply: pick the tool call in the agent's toolkit that talks to the least reliable external dependency, and ask what happens if that dependency starts returning 429s for thirty seconds straight. If the answer is "the agent keeps calling it at the same rate until the turn budget runs out," there's no backoff. If the answer is "the agent's other sub-agents keep calling it just as fast because they don't share that information," there's backoff without shared state. Either gap turns a normal, recoverable rate limit into the reason the whole run failed.
Related: a bad canary slice failing loudly and fast is exactly the kind of event a circuit breaker should contain automatically — see canary and shadow rollouts for agent changes for the rollout side of that same instinct.
A shared rate limit is also one of the easiest ways for tool calls that look independent to interfere with each other — see parallel tool calls for agents for how a parallel batch can trip a limit that no single call in it would have hit alone.