Build a 40-email eval before you write an email-routing agent
An enum of team labels, one pytest file and 40 real emails give you the numbers to defend a router design in front of an interviewer.
In brief
- Routing means classify, then forward. Start with a simple classifier, not a multi-step agent.
- Force the label into an enum with Structured Outputs, and treat a refusal as a signal to send the email to a person.
- The 40 samples must match the real email distribution and include ambiguous cases. Score with string matching, then read each team's score separately.
Asked to design a system that routes email to the right team, many candidates immediately draw a multi-step agent with memory, tools and a self-correcting loop. A stronger answer goes the other way: a simple classifier, plus a number that proves it works.
In “Building effective agents”, Anthropic describes routing as classifying an input and directing it to a specialized downstream task.
The same piece advises adding complexity only when it demonstrably improves outcomes. So this guide builds two things together: a router, and a 40-sample eval to measure it.
For an FDE this is more than an interview exercise. Take a client’s shared inbox, a support@ or contact@ address where email for every team lands in one place. If you are asked to automate it, the engineer who brings a per-team scorecard to the demo will win the client’s trust more easily than one who brings only a polished demo.
What you will build, and what you need
You will end up with three files: router.py, which calls the model and returns a team label; evals/routing.jsonl, which holds 40 labeled emails; and evals/test_routing.py, which scores them automatically. You need Python, pytest and API access to a model that supports schema-constrained output.
The model call below is simplified. The function call_llm_structured is a placeholder: replace it with your provider’s real Structured Outputs call, following their documentation.
Step 1: the team label is an enum, not free text
The most common mistake is letting the model answer with something like “This email probably belongs to the accounting team.” Downstream code then has to extract the label, and every slight variation in wording is a bug. Lock the list of labels from the start:
TEAMS = ["billing", "tech_support", "sales", "legal", "human_triage"]
ROUTE_SCHEMA = {
"type": "object",
"properties": {
"team": {"type": "string", "enum": TEAMS},
"reason": {"type": "string"},
},
"required": ["team", "reason"],
"additionalProperties": False,
}
OpenAI’s documentation draws a clear line between two modes: JSON mode does not guarantee the output matches a schema; only Structured Outputs does. It also says Structured Outputs removes the need to validate or retry when the model returns a malformed response.
The human_triage label is your own design choice and comes from no documentation. It is the exit for when the model is unsure, and worth having an explanation ready for if you are asked.
Check: print ROUTE_SCHEMA and read it back. Every team at the client should have exactly one label, with no two labels meaning the same thing.
Step 2: the prompt defines the teams, the code handles refusals
SYSTEM_PROMPT = """You classify emails sent to a shared inbox.
billing: invoices, payments, refunds.
tech_support: product bugs, login, integrations.
sales: quotes, upgrades, trials.
legal: contracts, personal data, legal complaints.
human_triage: belongs to no team, or equally to several teams."""
def route(email_text: str) -> dict:
# simplified: replace with a real Structured Outputs call
resp = call_llm_structured(SYSTEM_PROMPT, email_text, ROUTE_SCHEMA)
if resp.refusal:
return {"team": "human_triage", "reason": "model refused"}
return resp.parsed
OpenAI says that with Structured Outputs, a safety-based refusal from the model can be detected programmatically. A router should not let those emails disappear. Sending them to a person is the safest option.
Keeping the team definitions in the prompt pays off when working with a real client. Anthropic’s ticket-routing guide points out that the model can adapt when category definitions change or new categories are added, without extensive relabeling of training data.
When the client restructures and merges sales with billing, you edit a few lines of the prompt and rerun the eval.
Step 3: the 40 samples must look like the real inbox
Anthropic’s eval documentation asks that an eval reflect the real distribution of the task and not forget edge cases.
Its ticket-routing guide argues that a few dozen labeled examples are enough for the model to classify well. That refers to the data given to the model, not to the size of the eval set. The figure of 40 here is a pragmatic choice: enough to see errors repeating within each team, few enough to label yourself in an afternoon.
Consider the inbox of a hypothetical SaaS company. One reasonable split, offered only as a suggestion, is 30 typical emails divided in the real proportions across teams, 6 ambiguous emails and 4 that are spam or off-topic.
Anthropic counts among edge cases the ambiguous ones that even humans would struggle to agree on. An example is an email saying “this month’s invoice is wrong because the report export feature is broken”, which is both a billing issue and a technical one.
{"id": "e07", "email": "The October invoice charged us twice...", "expected": "billing", "tag": "typical"}
{"id": "e33", "email": "Invoice is wrong because the export is broken...", "expected": "human_triage", "tag": "ambiguous"}
For ambiguous cases, the right label does not appear on its own. You have to agree a rule with the client: send to a person, or send to the team with primary responsibility. Write that rule into the prompt; otherwise the eval will penalize the model for a decision nobody has made.
Step 4: score with assertions, read the score by team
In “Your AI Product Needs Evals”, Hamel Husain calls the first level of evals unit tests for LLMs: assertions of the kind you already write in pytest. For a router, the assertion is just a string match between the predicted label and the expected one.
import json
import pytest
from router import route
CASES = [json.loads(line) for line in open("evals/routing.jsonl")]
@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_route(case):
assert route(case["email"])["team"] == case["expected"]
pytest evals/test_routing.py -q
Anthropic also recommends many automatically graded questions, even with a slightly weaker signal, over a few graded by hand. So in the first round, do not replace assertions with reading every answer yourself.
Now run a calculation with illustrative numbers. The 40 samples are split as in Step 3, with the 6 ambiguous cases and the 4 spam emails all expected to go to human_triage. The router passes 36 of 40, or 90%. That sounds fine until you break the score down by team:
| Team (illustrative) | Samples | Correct | Rate |
|---|---|---|---|
| billing | 10 | 10 | 10/10 |
| tech_support | 9 | 9 | 9/9 |
| sales | 6 | 6 | 6/6 |
| legal | 5 | 1 | 1/5 |
| human_triage | 10 | 10 | 10/10 |
| Total | 40 | 36 | 90% |
All four failures fall in legal. The router is missing 4 of 5 legal emails, the very kind that costs most when it arrives late.
Check: group results by expected and by tag. If ambiguous cases fail often while typical cases are almost all correct, fix the rule in the prompt; the architecture can stay as it is.
When to add complexity
The answer is in the scorecard. With the table above, the first job is to fix exactly where the router fails: rewrite the legal definition in the prompt more clearly, for instance by listing the kinds of legal email that are often misrouted, then rerun all 40 samples.
If the legal score does not rise, the change is not worth keeping, however sensible it looks. That discipline is worth showing plainly in an interview.
Team-specific prompts are a separate step that comes later. Anthropic explains why routing helps: without separate paths, optimizing for one kind of input can hurt performance on others.
So when you want the system to do more, such as drafting replies for billing, put that prompt behind the router rather than piling more rules into the classification prompt.
A drafting prompt does not make the router pick teams more accurately, so it needs its own eval and cannot rely on the score from these 40 samples.
Once in production, Husain warns, you must never stop looking at the data. Every misrouted email that the receiving team reports should become a new line in routing.jsonl. The 40-sample set will grow to 60, then 100.
Taking the skill into interviews and onto your CV
In a system design round, spend the first minute saying how you will measure: the label enum, the human_triage fallback, an eval with ambiguous cases and per-team scores. Only then draw boxes and arrows. If you are asked “what if the company reorganizes its teams?”, the answer is ready: edit the definitions in the prompt and rerun the eval.
On a CV, a line such as “built a 5-label email router with a 40-sample eval, raising legal recall from 1/5 to 5/5” is far more convincing than “experience with LLM agents”. The numbers in that example are illustrative, but keep the structure: it shows you know how to prove things with numbers before asking for more complexity.
Was this article useful?
Thanks for the feedback!