# An agent that says it cancelled the order may not have

> A τ-bench-style grader checks the database rather than the transcript, and takes an afternoon of Python on a laptop to build.

Original: https://fdetimes.net/en/guides/stateful-agent-evaluation-tau-bench-harness/

The agent tells the customer: "I've cancelled your order." The transcript looks perfect. Whether the order has actually moved to a cancelled state in the database is another matter.

Anthropic's guide to evals makes exactly this point with a booking example: the real outcome is whether the environment's SQL database contains that booking row, and an agent can perfectly well report success while the actual state says the opposite.

A cancelled order is no different: what needs checking is the order's row in the data, not what the agent said.

τ-bench, a benchmark published by Sierra in 2024, is built around that idea. It ignores the dialogue and looks only at the data: when the conversation ends, the database is compared with a goal state that the benchmark's authors labelled in advance. Sierra calls this stateful evaluation, and it removes the need for a human or an LLM to do the grading.

The skill matters because at a customer site the question is always "did the agent do the right thing?", not "did the agent say it did?". This guide builds a miniature grader using the same mechanism as τ-bench, runnable on a laptop with nothing but Python.

## What will you build, and what do you need?

The end product is a `reward` function that gives each run a score of 1 or 0, plus a function that computes pass^k across many runs. You need Python 3, a "database" in the form of a dict, and an agent that can call tools. Any agent framework will do, as long as it changes data through tool functions.

All the code below is a simplified version, rewritten to illustrate the mechanism. It is not code from the τ-bench repo. The original lives in `tau_bench/envs/base.py` and is worth reading once you have finished.

## Step 1: one hash function for the whole database

In the reference implementation, the `get_data_hash` function hashes all of the environment's data as soon as the agent finishes. The simplest equivalent is to serialise the dict with a fixed key order and then hash it:

```python
import copy, hashlib, json

def data_hash(data: dict) -> str:
    blob = json.dumps(data, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()
```

How to check it: call the function twice on the same dict, then on a `deepcopy`. All three calls must return the same string. If they differ, something in your data is non-deterministic, usually a set or an object that does not serialise stably.

## Step 2: write each task as a sequence of correct actions

In τ-bench, the agent has tool APIs and a set of policies for each domain, while the user is played by an LLM and converses over several turns. Each task comes with its answer in the form of correct actions, not a model reply. Picture a retail shop:

```python
def cancel_order(data, order_id, reason):
    data["orders"][order_id]["status"] = "cancelled"
    data["orders"][order_id]["cancel_reason"] = reason

TOOLS = {"cancel_order": cancel_order}

task = {
    "instruction": "Customer wants to cancel order W1001 because they ordered by mistake",
    "actions": [("cancel_order",
                 {"order_id": "W1001", "reason": "ordered by mistake"})],
    "outputs": [],
}
```

How to check it: run the actions in `actions` by hand on a copy of the data and inspect the result. It must be exactly the state a good support agent would leave behind after this case.

## Step 3: compare the agent's hash with the answer's hash

This is the core. τ-bench's `calculate_reward` function replays the correct actions on a clean copy of the data, compares the resulting hash with the agent's final hash, and gives a score of 0 if they differ.

The reference implementation also checks `task.outputs`: if a task requires the agent to state a specific string, that string is compared case-insensitively with commas removed, and missing it costs the point.

```python
def reward(initial, agent_data, task, agent_replies):
    gt = copy.deepcopy(initial)
    for name, args in task["actions"]:
        TOOLS[name](gt, **args)
    if data_hash(agent_data) != data_hash(gt):
        return 0.0
    said = " ".join(agent_replies).lower().replace(",", "")
    for out in task["outputs"]:
        if out.lower().replace(",", "") not in said:
            return 0.0
    return 1.0
```

Now try a hypothetical case. The agent cancels the right order, W1001, but records the reason as "no longer needed", when the customer clearly said they ordered by mistake. The transcript still reads well and the customer still says thank you, but the `cancel_reason` field differs, so the hash does not match and the run scores 0. A human grader skimming the transcript could easily miss this error.

**Key point:** What the agent says is only testimony. The state of the database is the evidence.

## Step 4: run it many times and compute pass^k

Getting it right once does not tell you much. Sierra introduced pass^k to test consistency: on the same task, can the agent repeat the correct result?

As Anthropic explains it, this is the probability that none of k runs fails, so the number falls as k grows, which is why it matters particularly for agents that deal directly with customers.

```python
def pass_hat_k(results):  # {task_id: [1.0, 0.0, ...]} with k runs per task
    ok = [all(r == 1.0 for r in runs) for runs in results.values()]
    return sum(ok) / len(ok)
```

This function is a simplification: it runs exactly k times per task and counts a task as passed only when all k runs succeed. If you want to compare your numbers with those in the τ-bench paper, read how the paper computes them first.

Try the arithmetic: if an agent succeeds 60% of the time per run and the runs are independent, the probability of succeeding on all 8 runs is 0.6 to the power of 8, about 1.7%.

The 60% average looks respectable, but what customers experience is closer to 1.7%.

τ-bench's real results point the same way. Even the best GPT-4o agent succeeds on fewer than half of the tasks, averaged across both domains, and in retail, pass^8 for strong function-calling agents is below 25%. Sierra writes that every agent tested performed very poorly on the consistency test.

## What if you want to run the original?

The `sierra-research/tau-bench` repo runs through a CLI in which you choose the agent strategy, environment and model:

```bash
python run.py --env retail --model gpt-4o --agent-strategy tool-calling
```

The repo also lets you run individual tasks by ID; the exact syntax is in the README. Note that the repo states clearly that the task set is no longer being updated. Use it to learn the grading mechanism, not as a current yardstick for a customer's agent.

## Mistakes that make the grader fool you

**Hashing random fields.** If timestamps, randomly generated IDs or update times end up in the hash, an agent that did the right thing can still score 0. Strip these fields out or normalise them to a fixed value before hashing, and record which fields you excluded.

**Ambiguous correct actions.** If a customer request can be handled in two ways that both comply with policy, an exact hash comparison will unfairly penalise the agent. Rewrite the instruction so that only one correct goal state remains.

**Forgetting `outputs` on read-only tasks.** If the customer asks for their balance, the database does not change, so the hash will match even when the agent reads out the wrong figure. For this kind of task, the required string in the reply is what decides the score.

There is one more temptation: going back to the transcript to "give credit for effort". Anthropic advises grading what the agent produced, not the path it took. Use transcripts to find the cause when a task fails, not to grade.

## How does this skill show up at a customer site?

In a real deployment, the "database" is a sandbox of the customer's CRM, ticketing system or booking system. The hard part is not the code but sitting down with domain experts to write the goal state for each scenario, including scenarios where policy forbids the agent from acting.

The first thing to do on arrival is to ask for a copy of the data that can be reset, because without a reset you cannot rerun, and without reruns there is no pass^k.

On an FDE CV or in an interview, a line such as "built an eval harness that grades agents by database state, reporting pass^k on tasks drawn from real tickets" carries far more weight than "experience with LLMs". When a job description mentions evals or reliability, have the story of a grader like this ready to tell in the interview.

Next time you demo to a customer, don't just show them the conversation. Show them two matching hashes, eight times in a row.

**Try this week:**

- Pick 5 real tasks from a customer's system or your own project, write the correct actions for each and grade the agent with hashes instead of reading logs
- Run each task 8 times and report both the average success rate and pass^8, then see how far apart the two numbers are
- Clone the sierra-research/tau-bench repo, run run.py with the retail env and read the calculate_reward function in tau_bench/envs/base.py

## Sources

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://arxiv.org/abs/2406.12045)

- [sierra-research/tau-bench (GitHub)](https://github.com/sierra-research/tau-bench)

- [τ-bench: Shaping the development and evaluation of agents (Sierra)](https://sierra.ai/blog/benchmarking-ai-agents)

- [tau_bench/envs/base.py (sierra-research/tau-bench)](https://github.com/sierra-research/tau-bench/blob/main/tau_bench/envs/base.py)

- [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
