# Generate 96 test cases with an LLM, keep the 50 you can trust

> Getting an LLM to write test questions is fast. The hard part is throwing out the cases you could not grade yourself.

Original: https://fdetimes.net/en/guides/generate-llm-test-cases-keep-trustworthy/

In a few minutes an LLM can write hundreds of test questions for your agent. The trouble is that many of them look alike, and for some of them even you are not sure what a correct answer looks like.

In its guide to evals for agents, Anthropic says 20-50 simple tasks drawn from real failures are already a good starting point. Producing lots of questions is the easy part. The hard part is choosing the few dozen cases you are willing to use as a benchmark.

This guide goes from a list of dimensions to 96 traces, then filters them by hand down to 50 cases. You need Python 3, an agent that runs locally (or on staging), access to an LLM, and the customer's real business documents. The code below is **simplified**: `call_llm` and `run_agent` are wrappers you write yourself, not functions from any library.

## Step 1: Don't ask the LLM for questions straight away

Hamel Husain and Shreya Shankar recommend generating synthetic questions from tuples of dimensions, where each tuple takes exactly one value from each dimension. Picture a customer-service agent for a delivery company, with four dimensions: intent, customer type, how complete the information is, and whether the agent should act or refuse.

```python
import itertools, json

DIMENSIONS = {
    "intent": ["track_order", "change_address", "refund"],
    "persona": ["new_customer", "angry_customer"],
    "info": ["complete", "missing_order_id"],
    "expectation": ["should_act", "should_refuse"],
}
tuples = [dict(zip(DIMENSIONS, combo))
          for combo in itertools.product(*DIMENSIONS.values())]
print(len(tuples))  # 3 * 2 * 2 * 2 = 24
```

**Check:** you have exactly 24 tuples, and each line tells you at a glance what that case tests. If some combinations make no sense for the real business, delete them now. Dropping a question after it has been written costs far more effort. The example keeps all 24 tuples to keep the arithmetic simple. If you delete some, raise the number of questions per tuple so the set stays at around 100.

The `expectation` dimension is there for a reason. Anthropic warns that a one-sided eval pushes you to optimise the agent in one direction only. If every case expects a refund, you end up rewarding an agent that hands out refunds indiscriminately.

## Step 2: Write the questions in a separate call, grounded in real documents

Hamel and Shreya explain that separating tuple generation from question writing keeps the questions from repeating the same phrasing. DeepEval's documentation treats document-based generation as the strongest default, because the questions stay anchored to your knowledge base.

At the other extreme, letting the LLM make up questions with no documents at all is the least grounded approach, and suits only simple problems.

```python
policy = open("refund_policy.md").read()  # the customer's real document

def write_queries(t, n=4):
    prompt = (
        f"Using only this policy:\n{policy}\n\n"
        f"Write {n} different customer messages for: {json.dumps(t)}. "
        "One message per line. Vary wording and length."
    )
    return call_llm(prompt).splitlines()  # simplified wrapper

cases = []
for i, t in enumerate(tuples):
    for j, q in enumerate(write_queries(t)):
        cases.append({"id": f"c{i}_{j}", **t, "query": q})
print(len(cases))  # 24 * 4 = 96
```

**Check:** skim 10 random questions. If five of them start the same way, go back and confirm the prompt is actually receiving the tuple.

## Step 3: Run the agent once per case, save the traces and read them

In this guide, a trace is one agent run on one case. At this step each case runs only once, so 96 cases give exactly 96 traces, and each trace carries its case's id. According to Hamel and Shreya, about 100 diverse traces make a useful starting set for finding failures.

```python
with open("traces.jsonl", "w") as f:
    for c in cases:
        out = run_agent(c["query"])  # simplified wrapper
        f.write(json.dumps({**c, "trace": out}) + "\n")
```

**Check:** the file has 96 lines. Open it and read the traces yourself rather than just counting lines. This is when you note down the first failures.

## Step 4: A case is trustworthy only when two graders agree

This step decides whether the test set can be trusted. Hamel and Shreya advise annotating at least 30 traces yourself rather than relying on automated suggestions. DeepEval also says generated goldens must be reviewed, corrected and extended before you treat them as ground truth.

The clearest filtering rule comes from Anthropic: a good task is one where two domain experts, grading independently, would reach the same pass or fail verdict.

So you grade the 96 traces in one file, someone who knows the customer's business grades them in another, and neither of you sees the other's work. Because each case has only one trace, grading by id covers both the case and its trace.

```python
import csv

def load(path):
    return {r["id"]: r["verdict"] for r in csv.DictReader(open(path))}

a, b = load("labels_me.csv"), load("labels_ops.csv")
agreed = [i for i in a if i in b and a[i] == b[i]]
disputed = [i for i in a if i in b and a[i] != b[i]]
print(len(agreed), len(disputed))
```

Imagine the result: 96 traces, and the two graders disagree on 30. Don't rush to delete all 30.

Each disputed case points to a policy question the customer has not answered clearly. For example: can an angry customer get a refund for an order that arrived one day late?

Keep the cases you can fix by clarifying the question. Drop the ones that are ambiguous at the root.

**Key point:** If two people who know the business grade a test case differently, it is not ready to be a benchmark.

## Step 5: Rerun several times, cut to 50 cases and keep both sides balanced

Only now do you run repeatedly. For the agreed cases, run the agent three more times each. Anthropic stresses that you only know whether a grader is doing its job when you read the transcripts and scores of many runs.

```python
with open("trials.jsonl", "w") as f:
    for c in cases:
        if c["id"] not in agreed:
            continue
        for trial in range(3):
            out = run_agent(c["query"])  # simplified wrapper
            f.write(json.dumps({"id": c["id"], "trial": trial, "trace": out}) + "\n")
```

Look for the cases the agent failed all three times. According to Anthropic, a task that never passes usually signals a broken task, and only rarely a weak agent. Reread the task: the question may ask for information that does not exist in the policy document.

Then count by the `expectation` dimension. A sensible target is about 25 cases where the agent should act and 25 where it should refuse, with every intent represented. Write these 50 cases to `golden.jsonl`, along with the agreed pass/fail verdict and a one-line explanation of why.

## Common mistakes

The costliest mistake is reporting the failure rate on the synthetic set as if it were the real error rate. Hamel and Shreya say plainly that synthetic data does not tell you how common a failure is in production. If the agent fails 10 of 50 cases, you know it has failure modes worth fixing. You do not know that 20% of customers will hit an error.

Another mistake starts with a prompt that has no document attached. Suppose you ask the LLM to write refund questions without pasting in the policy: it may assume a 30-day refund window when the customer only allows 7 days.

An agent that follows the real policy then gets marked as failing. That is exactly the risk that leads DeepEval to rank generation from scratch as the least grounded approach.

A one-sided test set is hard to spot because the score looks great. If all 50 cases involve customers who qualify for a refund, an agent that refunds every request will score 50 out of 50. You only find out it is wrong when it meets an ineligible customer in real life.

The last mistake lives in the automated grader. Suppose the grader passes every answer containing the word "refunded", even when the agent refunded an order the policy does not allow. The overall score still looks good, and the problem only surfaces when you open the transcripts of many runs and read them, as Anthropic advises.

## Where does this skill show up in customer work?

On a customer site you often have to demo the agent before there are enough real logs. These 50 cases give you something to measure from the first week. The joint grading session with the customer's operations team is in effect a customer discovery session: every case the two sides grade differently is a business rule nobody has written down.

Once real traffic arrives, add cases drawn from real failures to the test set, since that is exactly the kind of task Anthropic considers a good starting point.

If you want to put this skill on your CV, describe it in terms of process and numbers: built a 50-case eval set from 96 traces, with two independent graders, balanced across both sides.

An FDE hiring manager reading that line will see that you know how to measure, not just how to demo.

The final output is the `golden.jsonl` file. But the cases you struck out, and the reasons you struck them, are what the customer most needs to read, because they record the business rules nobody had written down.

**Try this week:**

- Pick an agent you are working on, write down 3-4 dimensions with their values, and count how many tuples you get
- Ask a colleague to grade 30 traces independently alongside you, then count the cases where your verdicts differ
- Filter out the cases the agent fails on every run and reread the task before blaming the agent

## Sources

- [Q: What is the best approach for generating synthetic data?](https://hamel.dev/blog/posts/evals-faq/what-is-the-best-approach-for-generating-synthetic-data.html)

- [Demystifying evals for AI agents](https://anthropic.com/engineering/demystifying-evals-for-ai-agents)

- [Introduction to Synthetic Data Generation (DeepEval docs)](https://deepeval.com/docs/synthetic-data-generation-introduction)