Hands-on: building a 50-case eval for a Q&A assistant in Python

According to Anthropic Engineering, 20-50 simple tasks taken from real failures are already a very good start for an eval set. You don’t need thousands of cases, and you don’t need an expensive eval platform. So the bar for a first eval set is much lower than many people assume. The main cost is an afternoon of writing.

For an FDE, that afternoon is worth more than any demo. When a customer asks “is the new prompt better?”, “it seems better” will not convince anyone. A table of 50 questions with pass rates before and after the change will.

This guide walks step by step through building a 50-case eval in Python for an assistant that answers questions over internal documents. You need Python 3, pytest and an answer(question) function that calls your assistant. The code below is a cut-down teaching version, not a complete framework. The sample data comes from a Vietnamese deployment, so the questions and refusal phrases are in Vietnamese. Swap in your own language.

Step 1: What does “a good answer” mean?

Anthropic’s documentation recommends making success criteria specific. Its example is to replace “good performance” with “accurate sentiment classification”. Criteria must also be measurable, either through quantitative metrics or through a clearly defined qualitative scale.

For a Q&A assistant, don’t write “answers well”. Write down three criteria instead. For example: the answer contains the correct facts from the documents, it does not make things up when the documents lack the information, and it does not answer outside its scope. Each criterion must be answerable with yes or no.

Check after this step: give the three criteria to a colleague. If they ask “what counts as correct?”, your criteria are still vague.

Step 2: Where do the 50 cases come from?

The best source is real failures: logs, support tickets and user complaints. Anthropic stresses that an eval set should reflect the real distribution of tasks and should not leave out edge cases. Suppose 70% of real questions are about leave policy. An eval made up entirely of salary questions will not measure what users actually need.

The set also needs balance. Anthropic Engineering warns that one-sided evals lead to one-sided optimisation. If all 50 questions have answers, you will end up rewarding an assistant that always answers, whether or not it knows.

Here is one suggested split, which you should adjust to your own data: around 30 questions with a clear answer in the documents, 10 edge cases such as ambiguous questions, typos or mixed-in English, and 10 questions where the assistant must say it has no information.

Every case must pass Anthropic Engineering’s test: two domain experts, grading independently, must reach the same pass/fail verdict. If the two still disagree about a question, rewrite it or drop it.

Step 3: Store the cases as a JSONL file

Each line is one case. The grader field says whether the case is graded by code or by a model.

{"id": "q01", "question": "How many days of paid leave do full-time employees get?", "must_contain": ["12 days"], "grader": "code"}
{"id": "q41", "question": "Does the company support MBA tuition?", "expect_refusal": true, "grader": "code"}
{"id": "q47", "question": "Summarize the process for requesting unpaid leave", "reference": "Submit a request to your direct manager...", "grader": "model"}

The three questions ask how many days of leave permanent staff get (expected answer: “12 ngày”, 12 days), whether the company funds MBA tuition (the assistant should refuse), and for a summary of the unpaid-leave process (reference: “submit a request to your direct manager…”). Their content is hypothetical. Check after this step: the file has exactly 50 lines and no id appears twice.

Step 4: Use pytest for anything code can grade

Anthropic prefers questions that can be graded automatically, for example by string matching or code-based grading. Its reasoning is that many automatically graded questions, even if each is slightly less accurate, beat a few questions carefully graded by hand.

Hamel Husain also describes the first layer of LLM evals as simple assertions, much like writing pytest tests.

# test_eval.py — shortened version
import json
import pytest
from assistant import answer  # the function that calls your assistant

with open("eval_50.jsonl", encoding="utf-8") as f:
    CASES = [json.loads(line) for line in f if line.strip()]

CODE_CASES = [c for c in CASES if c["grader"] == "code"]
REFUSAL_MARKERS = ["no information", "not found"]

@pytest.mark.parametrize("case", CODE_CASES, ids=lambda c: c["id"])
def test_code_graded(case):
    out = answer(case["question"]).lower()
    if case.get("expect_refusal"):
        assert any(m in out for m in REFUSAL_MARKERS)
    else:
        for s in case["must_contain"]:
            assert s.lower() in out

The comments mark this as a cut-down version, with answer as the function that calls your assistant. The refusal markers mean “no information” and “not found”.

Run pytest test_eval.py -q. Each case appears as its own test, named by id, so when one fails you know at once which one it is. Every result is simply pass or fail. Husain has found fine-grained scoring scales much harder to manage than binary grading.

Step 5: Free-form answers need a judge

A question like “summarise the process” cannot be checked by string matching. Evidently AI’s LLM-as-a-judge guide uses a binary judge that marks each answer right or wrong and must give an explanation. Anthropic’s documentation also recommends grading with a different model from the one that produced the answer.

JUDGE_PROMPT = """Question: {q}
Reference answer: {ref}
Answer to grade: {out}
Does the answer match the reference answer in content?
Return JSON: {{"verdict": "PASS" or "FAIL", "reason": "one sentence"}}"""

def judge(case, out):
    prompt = JUDGE_PROMPT.format(q=case["question"], ref=case["reference"], out=out)
    raw = call_judge_model(prompt)  # placeholder: call a model DIFFERENT from the assistant's model
    return json.loads(raw)

The prompt gives the judge the question, the reference answer and the answer to grade. It asks whether the content matches the reference and requests JSON with a verdict of “PASS” or “FAIL” and a one-sentence reason. call_judge_model is a placeholder you write yourself with your provider’s SDK, and it must call a model other than the assistant’s. Check after this step: read the reason field for any 5 cases. Explanations that make no sense usually mean the judge prompt needs work.

Step 6: The judge needs grading too

This is the step people skip most often. Evidently AI’s method is clear: treat your hand labels as ground truth and the LLM’s labels as predictions, then compare the two as you would when evaluating a classifier.

human = ["PASS", "FAIL", "PASS", ...]   # graded by you
judged = ["PASS", "PASS", "PASS", ...]  # graded by the judge on the same questions
agree = sum(h == j for h, j in zip(human, judged)) / len(human)
false_pass = sum(h == "FAIL" and j == "PASS" for h, j in zip(human, judged))
print(agree, false_pass)

Here human holds your own labels and judged holds the judge’s labels for the same cases. Watch false_pass closely: wrong answers that the judge lets through are the most dangerous kind of error. To make hand grading fast, Husain stresses removing all friction from looking at the data. To get started, export the question, answer, verdict and reason to a CSV file and open it in a spreadsheet.

What does a prompt fix look like on the scoreboard?

Picture a hypothetical run. The original prompt scores 31/50. You add one instruction: if the documents don’t contain the information, say so plainly instead of inferring an answer. You rerun both parts, and the scoreboard looks like this (trước = before, sau = after, tổng = total):

# Hypothetical numbers, only to illustrate how to read the results
                   before   after
code  (40 cases)   26/40    31/40
model (10 cases)    5/10     7/10
total              31/50    38/50

The figures are hypothetical and only show how to read the table. Don’t rush to put 38/50 in your report. You hand-grade the 10 judge-graded questions again and find one false_pass:

q47  judge: PASS   you: FAIL
judge's reason: "Correctly states the step of submitting a request to the direct manager."
actual error: the answer invented an extra approval step that is not in the documents

The judge passed q47 because the answer “correctly states the step of submitting the request to the direct manager”. You failed it. The real error is that the answer invented an extra approval step that isn’t in the documents.

So the model-graded group is really 6/10, and the honest total is 37/50. The prompt fix is still a clear improvement. You have also learned something else: the judge prompt needs to tell it to check for extra details, not only for missing ones.

Three mistakes that make an eval useless

The first is making up questions instead of taking them from logs. Questions written by engineers tend to be cleaner and easier than what real users type, so the pass rate looks good but tells you nothing.

The second is leaving out questions the assistant should decline to answer. Without that group, every prompt change that makes the assistant “bolder” looks like progress. The third is using the assistant’s own model as the judge, and never comparing the judge with hand labels.

At the customer site, the eval set is a contract

In a real deployment, write these 50 questions with the customer’s people. They are the “domain experts” who have the standing to decide what passes. Once both sides agree on the question set, arguments like “the bot has got worse lately” become a shared look at which questions just went from pass to fail.

The three pieces in this guide are also the three kinds of grader that Anthropic Engineering says are usually combined: code, model and human. If you are aiming for an FDE role, put a small repo like this on GitHub. Add a README with the before-and-after scoreboard for one prompt change and the false_pass case you caught.

When reading job descriptions, search for phrases such as “evaluation” or “LLM-as-a-judge” to see whether the role needs this skill. If it does, put the repo first in the projects section of your CV.

The next time someone asks whether the new version is better, you won’t have to guess. Run the eval and you have the answer.