FDE Times dataFDE jobs open 924▲33 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 80Companies hiring 196Remote-friendly 25%Median US pay $200kTop hirer Databricks 107
中文VI

The newspaper of the Forward Deployed Engineer

Guides

Overseeing a client's AI agent does not mean approving every step

How to build risk flags and an approval queue so an agent pauses only where a human decision is actually needed, and resumes cleanly afterwards.

ShareLinkedInFacebookX

In brief

  • Guardrails handle the automated checks; humans approve only the decisions where a single mistake causes real damage
  • A paused run must be stored durably and resumed from its saved state, which is what lets the review queue be asynchronous
  • Guardrails protect only the tools they are attached to, so every path an agent can use to act must be inventoried
Table sorting the five tools of a hypothetical refund agent into three tiers. get_order and close_case run automatically; email_customer and refund need human approval (highlighted); run_sql is blocked outright.
Only commands where a single mistake does real damage join the human approval queue. Automated guardrails handle the rest of the checking. Hypothetical example from the article. Source: the article's illustrative policy table.

Anthropic’s published data on how people supervise agents contains a detail that sounds backwards at first. New users, with fewer than 50 sessions, switch on full auto-approve about 20% of the time; past 750 sessions, that share exceeds 40%. Yet the same experienced group interrupts the agent more often, not less.

The two are different behaviors. Experienced users stop clicking “approve” on every step, but they know exactly when to step in. Anthropic’s conclusion is that effective oversight does not require approving every action.

For a forward deployed engineer this paradox is a daily design problem. Once an agent is wired into a client’s real systems, reading orders, sending email and issuing refunds, the client asks two contradictory questions: how do we stop it doing something stupid on its own, and why does everything need a human to sign off? A well-built oversight layer answers both.

Machines check the cheap parts, people decide the expensive ones

OpenAI’s guidance on agents draws a clean line: guardrails are for automated checks, human review is for approval decisions. A guardrail is code. It answers questions with clear answers: does the amount exceed the order value, does the email leak internal data, are the arguments correctly formatted?

Reviewers are kept for questions that need judgment, where a single mistake can do real damage. Push questions a machine could answer into the queue and reviewers tire, then start approving just to clear it.

The list of tools that must pause belongs in a separate policy table, not mixed into the prompt. LangChain’s human-in-the-loop middleware works this way: you declare entries by tool name, and when a call matches an entry, execution is interrupted.

That is the risk flag. Anthropic adds two recommendations: let the agent pause to ask a human at checkpoints or when it gets stuck, and set conditions in advance under which it stops entirely.

When an agent pauses, what has to survive?

In OpenAI’s approval flow, a flagged call does not run, and the whole run pauses until you approve or reject it. Once a decision is made, you resume that same run from the saved state rather than starting a new conversation turn.

This small detail is what makes an asynchronous queue possible: a reviewer on the client side can answer after a meeting, and the agent still remembers what it was in the middle of.

The price is that state must live somewhere that survives a process restart. LangChain’s documentation states plainly that human-in-the-loop requires checkpointing to handle interrupts.

At a client site, if state lives only in memory, a redeploy or a pod restart wipes out everything awaiting review. Use a database the client’s operations team already backs up and monitors.

Following one case: a retailer’s refund agent

The following example is hypothetical, built to show how the pieces fit together. A customer-support agent has five tools, and the policy table sorts them into three tiers:

Tool Tier Automated guardrail What the reviewer can do
lookup_order Auto Reads only the requesting customer’s orders Not needed
close_ticket Auto Ticket must have a note Not needed
email_customer Review Contains no internal data Approve, edit, reject
issue_refund Review Amount ≤ order value Approve, edit, reject
run_sql Block Not applicable None

The core takes only a few dozen lines. The sketch below is framework-agnostic; the point is the order of the checks:

POLICY = {
    "lookup_order":   {"tier": "auto"},
    "close_ticket":   {"tier": "auto"},
    "email_customer": {"tier": "review", "decisions": ["approve", "edit", "reject"]},
    "issue_refund":   {"tier": "review", "decisions": ["approve", "edit", "reject"],
                       "guard": lambda args, order: args["amount"] <= order["order_total"]},
    "run_sql":        {"tier": "block"},
}

def handle_tool_call(run_id, state, call, order):
    rule = POLICY.get(call.name, {"tier": "block"})  # unknown tool: blocked by default
    if rule["tier"] == "block":
        return reject(call, "Tool is not in the policy")
    guard = rule.get("guard")
    if guard and not guard(call.args, order):
        return reject(call, "Over the limit, no human review needed")
    if rule["tier"] == "review":
        save_checkpoint(run_id, state)  # store durably, never just in memory
        return enqueue_for_review(run_id, call, rule["decisions"])
    return execute(call)

A customer writes in asking for a refund. In a single turn the agent proposes two calls: email_customer saying the refund has been made, then issue_refund for the full order value. Both pass the guardrails but sit in the review tier, so the run pauses, state is saved, and the queue shows the two actions in that order.

The reviewer opens the case and sees the order was already refunded through the payment gateway the day before. They edit the email to tell the customer to check their account, and reject the refund call with a reason. LangChain gives reviewers exactly these three options, and a reject decision refuses the call and sends feedback to the agent instead of executing it.

There is a trap here. In the default batched mode, decisions must be submitted in the same order as the actions in the interrupt request.

If your queue interface reorders them, say by moving money-related calls to the top for visibility, the email edit can end up applied to the refund call.

When the run resumes from state, the agent reads the rejection reason and closes the ticket with an accurate note.

Where to start at a client

The first job is to inventory every path the agent can use to act on the system. OpenAI’s documentation notes that tool guardrails run only on the function tools they are attached to. A general-purpose run_sql tool, an MCP server someone adds later, or a shell command in a sandbox are all back doors if they are missing from the table.

Next, sit down with the client’s process owner to assign tiers. The question to ask: if this call ran wrongly once, who would have to phone and apologize, and how long would it take to fix? The answer tells you the tier for each tool, and who the reviewer should be.

Only then write the guardrails, wire checkpoints to a durable database, and build a queue interface that shows enough context to decide on a single screen. Anthropic recommends extensive testing in a sandbox with appropriate guardrails before production, and this is where you replay real, anonymized cases to see which tiers pause too often.

The mistakes that turn a queue into a rubber stamp

The most common mistake is requiring too many approvals. When every call pauses, reviewers learn to approve without reading, and you get the feeling of safety without the substance. Track the share of calls approved unchanged for each tool; a tool that is almost always approved as-is is a candidate for the auto tier with a tighter guardrail.

The second is rejecting without explanation. An agent that does not know why it was blocked will propose nearly the same thing again, and the queue fills with the same request. The third is keeping state in memory, covered above, which usually surfaces on the very day the client redeploys.

On a CV, a line such as “designed a three-tier risk approval policy for a refund agent, with only X% of tool calls requiring human review” is far more persuasive than “experience with LLMs”.

The long-time users in Anthropic’s data approve less but interrupt more. The oversight layer you build for a client should help its reviewers do the same: fewer approvals, and timely interventions.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
4 sources
Read next on the roadmap · Stage 6: MeasurementThree metering bugs that turn token invoices into disputesTwo of them can be designed out of the usage event from the start; the third needs reconciliation finished before the invoice is finalized.