FDE Times dataFDE jobs open 873▼9 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 55Companies hiring 181Remote-friendly 25%Median US pay $200kTop hirer Databricks 98
中文VI

The newspaper of the Forward Deployed Engineer

Guides

An agent that is 90% right gets eight runs in a row right only 43%

Running 160 trials on a laptop is enough to tell whether a browser agent is reliable enough to hand over to a client.

ShareLinkedInFacebookX

In brief

  • If each run succeeds 90% of the time and runs are independent, the chance of getting all 8 right is only about 43%.
  • τ-bench found that even strong agents have a pass^8 below 25% in the retail domain.
  • A clean environment for every trial, grading on final state and traces switched on are the three conditions for a trustworthy number.
Slope chart showing per-run success rate and pass^8 after eight runs. 95% falls to 66%, 90% falls to 43% (highlighted), 80% falls to 17%.
Per-run accuracy is high, but the probability that all eight runs are correct (pass^8) is far lower, assuming runs are independent. Source: calculation in the article, based on the pass^k definition in Anthropic's article on agent evals.

Your browser agent has just filled in an order form in front of the client without a single stumble. But the client does not need it to work once. They need it to work on the 50th run, on a Monday morning, with nobody watching.

τ-bench introduced the pass^k metric to measure exactly this gap: how reliable an agent is when it runs the same task again and again. The results show that even strong function-calling agents such as gpt-4o succeed on fewer than 50% of tasks and are fairly inconsistent: pass^8 in the retail domain is below 25%.

This guide shows how to build that kind of benchmark on a laptop: 20 tasks, 8 runs each, 160 trials in total. At the end you will have three numbers and a list of failure types, enough to answer the question clients ask most: “Is this ready for real use?”

Why 90% is not good enough

The two metrics are often confused, so separate them first. pass@k counts a task as passed if at least one of k attempts is correct. pass^k, according to Anthropic’s article on agent evals, is the probability that all k attempts succeed.

Do the arithmetic. Assume a task has a fixed success rate per run and that runs are independent. Then pass^8 is that rate multiplied by itself 8 times, while pass@8 almost always reaches 100%:

Success rate per run pass@8 pass^8
80% close to 100% 0.8^8 ≈ 17%
90% close to 100% 0.9^8 ≈ 43%
95% close to 100% 0.95^8 ≈ 66%

An agent that is “right 9 times out of 10” sounds fine in a demo. Run it eight times in a row, though, and it completes the whole set less than half the time. Anthropic also notes that pass^k falls as k rises, and that it is the number that matters most for customer-facing agents.

One caveat should be stated plainly: real failures are rarely independent. They tend to cluster around the same pop-up or the same flaky step that breaks again and again, so 43% is an illustration, not a forecast. Your real number has to come from repeated runs, and from reading where the failures resemble each other, as step 5 does.

What you need

You need a working browser agent, built on any framework, and Playwright to record traces. You also need a test environment whose state you control, such as a staging copy of the client’s application or a sample app with its own database. Do not measure on production: you will run 160 trials, and many of them will fail.

Step 1: write 20 tasks and grade them on final state

Each task has two parts: the instruction given to the agent and a check function. The check function looks at the final state of the environment when the trial ends, not at the agent’s reply.

Take the task “change the shipping address on order #1042”. Do not grade it on the agent replying “Update complete”. Query the database to see whether the address field on that order has changed, and whether any other orders were touched. The snippet below is a simplified skeleton; db is a connection to your test environment:

def check_task_07(db):
    row = db.get_order(1042)
    return row["shipping_address"] == "12 Example St" and db.count_modified_orders() == 1

Pick tasks from the client’s real work, mixing easy ones with multi-step ones. Twenty tasks is not many, but it is enough to see which tasks are stable and which merely got lucky.

Check: run every check function on an untouched environment. All of them must return False. If one returns True from the start, that task is being graded wrongly.

Step 2: start every trial from a clean environment

Anthropic points out that agent behaviour varies between runs: a task that passes in one eval can fail in the next. An independent study of computer-use agents found the same even when the task and the model stayed fixed, and recommended evaluating agents through repeated runs.

For repeated runs to be comparable, each trial should start from the same state: if trial 3 leaves behind a cookie, a shopping cart or an edited record, trial 4 is being measured in a different world.

How you do this depends on your infrastructure. Typically it means restoring the database to a snapshot, using a fresh browser profile and logging in again with a test account. Write down anything that is hard to reset, because production will run into it too.

Check: run the same task twice in a row. If the second run is unusually fast or skips the login step, the environment is not clean.

Step 3: run 160 trials with tracing on

According to the Playwright documentation, the default configuration records a trace only on the first retry of a failed test. For this benchmark you need traces of every run, including successful ones, for comparison. If you wrap each trial as a Playwright test, you can force tracing on for every run:

npx playwright test --trace on

Alongside that, your harness writes each trial as one JSON line containing task_id, trial and passed (the result of the check function from step 1) to a file called results.jsonl.

Check: the file has exactly 160 lines, and each task_id appears exactly 8 times.

Step 4: compute three numbers from the same file

The script below uses only Python’s standard library:

import json
from collections import defaultdict

runs = defaultdict(list)
for line in open("results.jsonl"):
    r = json.loads(line)
    runs[r["task_id"]].append(r["passed"])

K = 8
tasks = list(runs.values())
pass1 = sum(sum(v[:K]) / K for v in tasks) / len(tasks)
pass_at_k = sum(any(v[:K]) for v in tasks) / len(tasks)
pass_hat_k = sum(all(v[:K]) for v in tasks) / len(tasks)
print(f"pass^1={pass1:.0%}  pass@8={pass_at_k:.0%}  pass^8={pass_hat_k:.0%}")

The script counts directly from the definitions: a task passes pass^8 when all 8 trials pass, and passes pass@8 when at least one trial passes. Picture a task that passes 7 of 8 trials. It counts in full for pass@8 but scores zero for pass^8.

The gap between pass@8 and pass^8 is your work list. The agent can already do those tasks; it just cannot yet do them consistently.

Step 5: read the traces of failed runs

The numbers tell you how often the agent fails. The traces tell you why. Playwright’s Trace Viewer is a graphical tool for opening and replaying recorded traces, so you can put a failed run next to a successful run of the same task and find the first step where the two diverge.

Anthropic treats reading transcripts as the way to find out whether your eval is measuring what it should.

Record each failure in a “failure type” column. It helps to know in advance what you are looking for. BrowserArena, testing agents on the live web, found three recurring failure types: solving captchas, closing pop-up banners and navigating directly to a URL.

Group failures by type to find recurring causes, then start fixing from the largest group. This is also where it becomes clear why the multiplication at the start is only an illustration: a single pop-up can break a whole batch of trials in the same way.

Step 6: degrade the environment on purpose

Your staging environment is cleaner than the real web. WAREX injected realistic instability into existing web-agent benchmarks and found that success rates fell significantly. ReliabilityBench likewise reported that input noise alone pulled the success rate from 96.9% at ε=0 down to 88.1% at ε=0.2.

You can build a simplified version: in the test environment, add a script that shows a cookie modal after a few seconds, or make an API return errors on a small share of requests.

Run the 160 trials again on the degraded environment and compare pass^8 with the previous round. The drop between the two is a measure of robustness that no demo will ever show you.

Three traps that make the numbers look falsely good

Do not grade on the agent’s own report: a line saying “done” does not prove the final state is correct, so always let the check function from step 1 decide.

Do not share one browser session across all 8 trials either, because later trials benefit from earlier ones. And do not report pass@k alone: by definition, it is always the highest of the three numbers.

Where this skill shows up on a client site

In an engagement, this three-number table is what you bring to the meeting that decides whether the agent goes to production. Instead of saying “the agent works pretty well”, you can say which tasks hit 8/8, which only reach 5/8, and why. That sentence helps the client decide at which step to keep a human reviewer.

On an FDE CV or in an interview, describe exactly this process: how many tasks, how many trials, and pass^k before and after you fixed the failure types. If a job posting mentions evals or reliability, this is the story to lead with.

The demo shows the client what the agent can do. The 160 runs show them what it will do once nobody is watching.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
7 sources
Read next on the roadmap · Stage 2: Broad engineeringWhen an ERP sync fails halfway, the safe retry is the hard partIdempotency keys, a transactional outbox and three-way reconciliation keep a client's books matching to the last cent when the period closes.