# Build a 40-email eval before you write an email-routing agent

> An enum of team labels, one pytest file and 40 real emails give you the numbers to defend a router design in front of an interviewer.

Original: https://fdetimes.net/en/guides/email-router-40-sample-eval/

Asked to design a system that routes email to the right team, many candidates immediately draw a multi-step agent with memory, tools and a self-correcting loop. A stronger answer goes the other way: a simple classifier, plus a number that proves it works.

In "Building effective agents", Anthropic describes routing as classifying an input and directing it to a specialized downstream task.

The same piece advises adding complexity only when it demonstrably improves outcomes. So this guide builds two things together: a router, and a 40-sample eval to measure it.

For an FDE this is more than an interview exercise. Take a client's shared inbox, a support@ or contact@ address where email for every team lands in one place. If you are asked to automate it, the engineer who brings a per-team scorecard to the demo will win the client's trust more easily than one who brings only a polished demo.

## What you will build, and what you need

You will end up with three files: `router.py`, which calls the model and returns a team label; `evals/routing.jsonl`, which holds 40 labeled emails; and `evals/test_routing.py`, which scores them automatically. You need Python, pytest and API access to a model that supports schema-constrained output.

The model call below is **simplified**. The function `call_llm_structured` is a placeholder: replace it with your provider's real Structured Outputs call, following their documentation.

## Step 1: the team label is an enum, not free text

The most common mistake is letting the model answer with something like "This email probably belongs to the accounting team." Downstream code then has to extract the label, and every slight variation in wording is a bug. Lock the list of labels from the start:

```python
TEAMS = ["billing", "tech_support", "sales", "legal", "human_triage"]

ROUTE_SCHEMA = {
    "type": "object",
    "properties": {
        "team": {"type": "string", "enum": TEAMS},
        "reason": {"type": "string"},
    },
    "required": ["team", "reason"],
    "additionalProperties": False,
}
```

OpenAI's documentation draws a clear line between two modes: JSON mode does not guarantee the output matches a schema; only Structured Outputs does. It also says Structured Outputs removes the need to validate or retry when the model returns a malformed response.

The `human_triage` label is your own design choice and comes from no documentation. It is the exit for when the model is unsure, and worth having an explanation ready for if you are asked.

**Check:** print `ROUTE_SCHEMA` and read it back. Every team at the client should have exactly one label, with no two labels meaning the same thing.

## Step 2: the prompt defines the teams, the code handles refusals

```python
SYSTEM_PROMPT = """You classify emails sent to a shared inbox.
billing: invoices, payments, refunds.
tech_support: product bugs, login, integrations.
sales: quotes, upgrades, trials.
legal: contracts, personal data, legal complaints.
human_triage: belongs to no team, or equally to several teams."""

def route(email_text: str) -> dict:
    # simplified: replace with a real Structured Outputs call
    resp = call_llm_structured(SYSTEM_PROMPT, email_text, ROUTE_SCHEMA)
    if resp.refusal:
        return {"team": "human_triage", "reason": "model refused"}
    return resp.parsed
```

OpenAI says that with Structured Outputs, a safety-based refusal from the model can be detected programmatically. A router should not let those emails disappear. Sending them to a person is the safest option.

Keeping the team definitions in the prompt pays off when working with a real client. Anthropic's ticket-routing guide points out that the model can adapt when category definitions change or new categories are added, without extensive relabeling of training data.

When the client restructures and merges sales with billing, you edit a few lines of the prompt and rerun the eval.

## Step 3: the 40 samples must look like the real inbox

Anthropic's eval documentation asks that an eval reflect the real distribution of the task and not forget edge cases.

Its ticket-routing guide argues that a few dozen labeled examples are enough for the model to classify well. That refers to the data given to the model, not to the size of the eval set. The figure of 40 here is a pragmatic choice: enough to see errors repeating within each team, few enough to label yourself in an afternoon.

Consider the inbox of a hypothetical SaaS company. One reasonable split, offered only as a suggestion, is 30 typical emails divided in the real proportions across teams, 6 ambiguous emails and 4 that are spam or off-topic.

Anthropic counts among edge cases the ambiguous ones that even humans would struggle to agree on. An example is an email saying "this month's invoice is wrong because the report export feature is broken", which is both a billing issue and a technical one.

```json
{"id": "e07", "email": "The October invoice charged us twice...", "expected": "billing", "tag": "typical"}
{"id": "e33", "email": "Invoice is wrong because the export is broken...", "expected": "human_triage", "tag": "ambiguous"}
```

For ambiguous cases, the right label does not appear on its own. You have to agree a rule with the client: send to a person, or send to the team with primary responsibility. Write that rule into the prompt; otherwise the eval will penalize the model for a decision nobody has made.

## Step 4: score with assertions, read the score by team

In "Your AI Product Needs Evals", Hamel Husain calls the first level of evals unit tests for LLMs: assertions of the kind you already write in pytest. For a router, the assertion is just a string match between the predicted label and the expected one.

```python
import json
import pytest
from router import route

CASES = [json.loads(line) for line in open("evals/routing.jsonl")]

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_route(case):
    assert route(case["email"])["team"] == case["expected"]
```

```bash
pytest evals/test_routing.py -q
```

Anthropic also recommends many automatically graded questions, even with a slightly weaker signal, over a few graded by hand. So in the first round, do not replace assertions with reading every answer yourself.

Now run a calculation with illustrative numbers. The 40 samples are split as in Step 3, with the 6 ambiguous cases and the 4 spam emails all expected to go to `human_triage`. The router passes 36 of 40, or 90%. That sounds fine until you break the score down by team:

| Team (illustrative) | Samples | Correct | Rate |
|---|---|---|---|
| billing | 10 | 10 | 10/10 |
| tech_support | 9 | 9 | 9/9 |
| sales | 6 | 6 | 6/6 |
| legal | 5 | 1 | 1/5 |
| human_triage | 10 | 10 | 10/10 |
| **Total** | **40** | **36** | **90%** |

All four failures fall in legal. The router is missing 4 of 5 legal emails, the very kind that costs most when it arrives late.

**Key point:** A router's average score can hide the one team the client is most afraid of missing.

**Check:** group results by `expected` and by `tag`. If `ambiguous` cases fail often while `typical` cases are almost all correct, fix the rule in the prompt; the architecture can stay as it is.

## When to add complexity

The answer is in the scorecard. With the table above, the first job is to fix exactly where the router fails: rewrite the legal definition in the prompt more clearly, for instance by listing the kinds of legal email that are often misrouted, then rerun all 40 samples.

If the legal score does not rise, the change is not worth keeping, however sensible it looks. That discipline is worth showing plainly in an interview.

Team-specific prompts are a separate step that comes later. Anthropic explains why routing helps: without separate paths, optimizing for one kind of input can hurt performance on others.

So when you want the system to do more, such as drafting replies for billing, put that prompt behind the router rather than piling more rules into the classification prompt.

A drafting prompt does not make the router pick teams more accurately, so it needs its own eval and cannot rely on the score from these 40 samples.

Once in production, Husain warns, you must never stop looking at the data. Every misrouted email that the receiving team reports should become a new line in `routing.jsonl`. The 40-sample set will grow to 60, then 100.

## Taking the skill into interviews and onto your CV

In a system design round, spend the first minute saying how you will measure: the label enum, the human_triage fallback, an eval with ambiguous cases and per-team scores. Only then draw boxes and arrows. If you are asked "what if the company reorganizes its teams?", the answer is ready: edit the definitions in the prompt and rerun the eval.

On a CV, a line such as "built a 5-label email router with a 40-sample eval, raising legal recall from 1/5 to 5/5" is far more convincing than "experience with LLM agents". The numbers in that example are illustrative, but keep the structure: it shows you know how to prove things with numbers before asking for more complexity.

**Try this week:**

- Take 40 real emails (with personal information removed) from a shared inbox, label each with a team and mark the ambiguous ones
- Write a test_routing.py file using pytest.mark.parametrize, run it and record each team's score
- Practice presenting this design in 5 minutes as a system design answer, opening with how you measure results

## Sources

- [Building effective agents (Anthropic Engineering)](https://www.anthropic.com/engineering/building-effective-agents)

- [Ticket routing (Claude Docs)](https://platform.claude.com/docs/en/about-claude/use-case-guides/ticket-routing)

- [Create strong empirical evaluations (Claude Docs)](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)

- [Structured Outputs (OpenAI API docs)](https://developers.openai.com/docs/guides/structured-outputs)

- [Your AI Product Needs Evals (Hamel Husain)](https://hamel.dev/blog/posts/evals/)
