← All packs

Designing when an agent should stop and ask a human

Permission scoping answers "is this agent allowed to send money, delete a branch, message a customer?" Escalation design answers a different question: of the things it's allowed to do, which ones should it still stop and ask about first? Those are separate lists, and conflating them produces one of two failure modes — an agent that asks about everything it's technically permitted to do, which trains the human to approve without reading, or an agent that only asks about things outside its permissions, which means the first time something goes wrong inside its own allowed scope, nobody heard about it until after.

Escalation is a trust budget, not a safety net

Every time an agent escalates something a human would have waved through in half a second, it spends down the credibility of the next escalation. A reviewer who has approved forty consecutive low-stakes prompts stops reading the forty-first, and the forty-first is precisely as likely to be the one that mattered as any of the others — escalation frequency doesn't correlate with escalation importance unless the system is designed to make it correlate. The goal isn't zero escalations and it isn't maximum coverage; it's reserving the interrupt for cases where a human's judgment actually changes the outcome, and handling everything else through a default the agent can act on alone.

Three questions before adding a gate

QuestionIf yesIf no
Is this action hard to reverse?lean toward escalatinglet the agent act, log it for later review
Would a human's answer actually differ from the agent's default?escalate — the human has information the agent doesn'ta gate here just adds latency, not judgment
Does this class of action recur often?batch it — one review of ten similar items, not ten interruptsa one-off gate is fine as a synchronous ask

An action that fails all three — reversible, no real judgment call, and rare — usually shouldn't have a gate at all. It's overhead pretending to be safety.

Batch the reviewable, interrupt for the irreversible

The instinct to escalate one item at a time feels safer but produces worse review quality: a human evaluating item seventeen of a live interrupt stream has no basis for comparison and no time to build one. Grouping same-class, reversible decisions into a single periodic batch — end-of-run summary, daily digest, one PR with ten similar diffs instead of ten separate approval requests — lets the human compare items against each other, which is often the only way to notice that one is different from the rest. Reserve the synchronous, blocking interrupt for the case that's actually time-sensitive and hard to undo: an irreversible external side effect, a spend past a hard ceiling, an action outside the scope anyone reviewed in advance. Not every gate needs to block; most should queue.

Pick a default for when nobody answers

An escalation with no timeout is a task that can hang forever waiting on a human who's asleep, in a meeting, or gone. Every gate needs an explicit answer to "what happens if no one responds in time," and the safe default is almost always to not act — hold the change, don't send the message, don't spend the money — rather than to proceed as if approved. A silent default-to-proceed converts every unanswered escalation into an unreviewed action, which defeats the reason the gate existed. The one exception worth designing for deliberately: a monitoring or health-check task where the safe default really is "do nothing and report," in which case a timeout isn't a fallback, it's the intended behavior.

Escalation copy matters as much as escalation logic. "Approve?" with no context forces the human to either dig for the details themselves or approve blind — both slower and worse than a request that states the action, the reason it triggered a gate, and what happens on both timeout and denial, in the first two lines.

Escalation design decides when to stop and ask. It assumes the underlying action was already inside the agent's allowed scope — for the separate question of what that scope should include in the first place, see permission scoping for agentic tools.