# Every call times out at 30 seconds. The user still waits 90

> Each step restarts the clock and retries push the worst case to 270 seconds; one deadline for the whole request, passed down, fixes it.

Original: https://fdetimes.net/en/guides/agent-timeout-budget-deadline-propagation/

A customer files a ticket: the support assistant your team has just deployed sometimes spins for 90 seconds before it reports an error. You open the code and find that every API call has a 30-second timeout, exactly as the checklist requires. No line is wrong, yet users wait three times longer than the number on paper.

This bug gets through code review because every line looks right. FDEs who put agents into production should understand it before go-live week, not after it shows up in a customer ticket. The cause is simple: each call has its own timeout, but users only care how long the whole request takes.

The fix needs no new infrastructure. It needs a different question about time: not "how long may each call wait?" but "how many seconds does the whole request have left?"

## Three 30-second waits make 90 seconds

Picture the assistant answering "where is my order?". The agent calls the CRM to find the customer, calls the order system for the status, then calls an LLM to write the reply. Each call has its own 30-second timeout.

One day the order system is slow. In the worst case all three steps hit their limit, and 30 plus 30 plus 30 makes 90 seconds.

Tianpan.co describes this mechanism in a piece on agent chains: the agent schedules the next tool call with a fresh timeout, as if the clock had just been reset.

Nothing in the chain tracks the total time spent.

Retries make it worse. If each step may be tried up to three times, the worst case is nine waits of 30 seconds, or 270 seconds. Those waits help no one either. Google's SRE book says it plainly: the client does not care what the server does after the deadline has passed.

## The deadline belongs to the request, not to each step

Google's SRE book says it is usually wise to set a deadline on an RPC, and goes a step further. A server should not invent a new deadline when it calls a backend. It should pass on the deadline it received. This is called deadline propagation.

The tianpan.co piece applies the same principle to agents. The deadline is set once as an absolute point in time, and that same point, not a duration, is passed to every later call.

**Key point:** The deadline belongs to the whole request, not to each call.

Why a point in time rather than a duration? A figure such as "20 seconds left" is correct only at the moment it is calculated. It does not account for time the request then spends in a queue or in the agent's reasoning loop. A deadline such as "expires at 10:00:30" is always correct, and every step can work out the remaining time for itself.

The gRPC documentation shows that both approaches exist in practice. Some APIs take a timeout (a duration), others take a deadline (a point in time), and only some implementations automatically pass the deadline of an incoming RPC on to outgoing RPCs. Check the library you use rather than assume it handles this for you.

## Example: adding a budget to the agent loop

Back to the support assistant. Suppose the customer accepts at most 30 seconds for a complete answer. The Python below is enough to fix the bug within a single process:

```python
import random
import time

class Budget:
    def __init__(self, seconds):
        # Set the deadline once; use the monotonic clock so system clock changes don't skew it
        self.deadline = time.monotonic() + seconds

    def remaining(self):
        return self.deadline - time.monotonic()

class TransientError(Exception):
    pass

def call_tool(fn, budget, reserve=2.0, max_attempts=2):
    for attempt in range(max_attempts):
        left = budget.remaining() - reserve  # keep time in reserve to reply to the user
        if left <= 0:
            raise TimeoutError("budget exhausted, not calling")
        try:
            return fn(timeout=left)  # each call gets only the time that remains
        except TransientError:
            wait = min(0.2 * 2 ** attempt, left) * random.random()
            time.sleep(wait)  # exponential backoff with jitter
    raise TimeoutError("out of attempts")

budget = Budget(30)
customer = call_tool(crm_lookup, budget)
order = call_tool(order_status, budget)
answer = call_tool(llm_summarize, budget)
```

Now replay the bad scenario. The CRM responds after 3 seconds and the order system hangs. The second call may wait only about 25 seconds: 30, minus 3, minus the 2-second reserve.

The example assumes that `order_status` raises its own client timeout error after 25 seconds. That error is not a `TransientError`, so `call_tool` does not catch it. It goes straight up the stack with no retry.

Even with a retry loop, `call_tool` would stop, because by then the remaining budget would be zero. The user gets an error at about the 28th second, not the 90th.

For agents, add one more step: check `budget.remaining()` before letting the LLM plan the next tool call. If less time remains than a step usually takes, return a partial answer with an explanation. That is better than starting work that cannot finish in time.

## Retries at every layer multiply the load

A retry gives the caller another chance to succeed, but the server has to spend more time on it. That is why the AWS Builders' Library calls retries "selfish". When each layer retries on its own, the load on the bottom layer multiplies. In the AWS example the load on the database rises 243-fold, and it will probably never recover.

Do the sum for your own system. Three layers, each trying three times, means one user request can become 27 calls to a slow service, just when it is at its weakest.

Google SRE advises limiting the number of retries per request and using exponential backoff with randomisation. The code above does both, and retries happen only within the budget.

## Applying this on a customer site

First, draw the call chain on paper. Write down each step's timeout and retry count, then multiply out the worst-case time. The number often surprises customers, and it is the best case for being allowed to fix the problem.

Next, ask the customer what the budget for the whole request should be. AWS acknowledges that choosing timeout values is often the hardest part, so the number should come from the user experience, not a guess.

Then set the deadline at a single entry point (the API gateway or the first handler) and pass it everywhere. Between services, use gRPC deadlines or a header carrying the point in time. Within a process, use an object like `Budget`. Finally, every step must check the budget before it starts expensive work.

## Common mistakes

The most common mistake is forgetting that the default may be infinite. The gRPC documentation states that gRPC sets no deadline by default, so a call can wait forever if the client does not set one. The second is passing "30 seconds" to every layer instead of the remaining time, which is the 90-second bug from the start of this article.

The third is harder to see: lower layers keep running after the layer above has given up. A database query that finishes at second 40 for a request that expired at second 30 is wasted resources.

The `Budget` class above only blocks further calls. You also need to pass `budget.remaining()` down as the timeout of the query itself or of the database driver, so that the lower layer cancels on its own when the request expires.

The fourth is leaving no reserve, so the deadline expires just as the reply still has to be formatted and sent.

## Turning the lesson into a line on your CV

Put this skill on your CV with a concrete number. A line such as "cut the worst-case wait of a customer support agent from 270 seconds to under 30 seconds with deadline propagation" carries far more weight than "experienced in reliability".

If an FDE job ad mentions "production agents" or "integration with customer systems", prepare this answer for the system design round. Be ready to walk through the 90-second sum and explain how you set a single deadline for the whole request.

A fast agent and a slow agent can run the same model and the same tools. What separates them is whether anything in the call chain tracks how many seconds the request has left.

**Try this week:**

- Pick an agent or service you are deploying. List every outbound call with its timeout and retry count, then multiply out the worst-case wait.
- Add a Budget class to the agent loop so it stops and returns a partial result when too little time remains for the next tool call.
- Find every gRPC client in the codebase that sets no deadline, and fix it.

## Sources

- [Addressing Cascading Failures (Google SRE Book, Chapter 22)](https://sre.google/sre-book/addressing-cascading-failures/)

- [Deadlines (gRPC guides)](https://grpc.io/docs/guides/deadlines/)

- [Timeouts, retries, and backoff with jitter](https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter)

- [Deadline Propagation in Agent Chains: What Happens to Your p95 SLO at Hop Three](https://tianpan.co/blog/2026/04/17/deadline-propagation-agent-chains-slo)