The LLM got it right. Your test still failed
String matching breaks on valid variations; asserting properties such as JSON structure and result count, in the pytest you already use, fixes it.
In brief
- String matching fails a test when the model gives the right answer in different words.
- Start with a structural test (valid JSON), then add a property for each scenario, such as the number of results returned.
- Run each test many times and agree an acceptable pass rate with the client.
The model returns {"listings":[{"id":42}]}, exactly the apartment the user wanted, and the test still fails because the reference answer has a space after each colon. The data is entirely correct. The test is what’s broken.
Developers fall into this trap easily. Unit tests usually have one correct answer, so comparing LLM output against a reference string seems like the obvious thing to do.
Anthropic’s engineers name the weakness of this approach directly: code-based graders are brittle when they meet valid variations that don’t match the expected pattern exactly.
You can fix it with tools you already use. Hamel Husain, writing about evals for AI products, defines unit tests for LLMs simply as assertions, the same kind you write in pytest. Keep the tool and change what you assert.
What you will build, and what you need
The running example comes from Husain’s article. It is a property-search feature: the user types a question in plain language and the LLM returns a list of properties.
You will write five tests across five steps. The first is a string-matching test, so you can see how it fails. The main scenario is that a user who searches for one exact address must get exactly one result back.
You need Python, pytest and any function that calls an LLM. The code below is trimmed down. Plug your real client into call_model.
# llm_app.py — trimmed version; replace call_model with a real client
def call_model(prompt: str) -> str:
raise NotImplementedError("Call your LLM here")
def search_listings(query: str) -> str:
prompt = ('Return JSON of the form {"listings": [...]} '
f"for the query: {query}")
return call_model(prompt)
Step 1: see why string matching fails
Anthropic’s documentation describes exact match the way many teams already use it. Output is compared with a predefined correct answer, usually after normalising whitespace and case. The test below does exactly that.
GOLDEN = '{"listings": [{"id": 42}]}'
def test_exact_match():
out = search_listings("apartment 12, A Street")
assert out.strip().lower() == GOLDEN.strip().lower()
strip() only trims whitespace at the two ends of the string. Spaces inside the JSON stay where they are, so the compact JSON from the opening still doesn’t match the reference answer.
Check: make call_model temporarily return '{"listings":[{"id":42}]}' (no spaces) and the test fails. It also fails once the model adds a "price" field on its own. Keep this test only as a reminder not to write tests this way again.
Step 2: test structure first, because it is cheapest
An analysis of property-based testing for LLMs on the tianpan.co blog says the easiest property to check is structure. The usual example is that the output must be valid JSON. This check needs no extra LLM call to grade it.
import json
def test_output_is_valid_json():
raw = search_listings("apartment 12, A Street")
data = json.loads(raw) # raises if the output is not JSON
assert "listings" in data
assert isinstance(data["listings"], list)
Check: make call_model return 'Found 1 apartment'. The test must fail, because json.loads raises an error. Switch to JSON with or without spaces and the test passes either way. Your suite no longer depends on how the string is formatted.
Step 3: assert properties for each scenario
Husain advises splitting the LLM’s scope into features and scenarios rather than using one check for everything. His property-search example needs a single line: len(listing_array) == 1.
def test_exact_address_returns_one_listing():
data = json.loads(search_listings("apartment 12, A Street"))
assert len(data["listings"]) == 1
The test ignores how the model describes the apartment. It asserts only what the user actually needs: a search for an exact address must not return five apartments, and it must not return an empty list. Other scenarios get other properties. A vague query, for instance, might need to return more than one result.
Step 4: one run proves nothing
A passing test in step 3 doesn’t settle the matter. Anthropic’s team warns that a task can pass one eval run and fail the next, because LLMs are non-deterministic. The tianpan.co article suggests running each test several times and requiring a minimum pass rate, for example 8 passes out of 10 runs.
RUNS = 10
MIN_PASS = 8
def passes_once(query: str) -> bool:
try:
data = json.loads(search_listings(query))
except json.JSONDecodeError:
return False
return len(data.get("listings", [])) == 1
def test_exact_address_pass_rate():
ok = sum(passes_once("apartment 12, A Street") for _ in range(RUNS))
assert ok >= MIN_PASS, f"only passed {ok}/{RUNS} runs"
Take an example. Out of ten runs, 7 return exactly one apartment, 2 return two apartments and 1 returns plain text instead of JSON.
That is 7/10, or 70%, which is below the threshold of 8. The test fails with the message “only passed 7/10 runs”. That number tells you far more than a flaky test that passes and fails at random with no explanation.
Check: run this test twice in a row against the real model. If the result moves around the threshold, you have measured how stable the feature really is.
Who decides on 8/10?
Engineers shouldn’t set this number alone. Husain writes that the pass rate is a product decision, and it depends on which failures you can accept. Traditional unit tests demand 100%. Here, the threshold reflects business needs. Returning one wrong apartment in a property search may be acceptable. Getting one clause wrong in a contract summary is not.
Don’t confuse the pass rate with pass@k either. According to Anthropic, pass@k measures the probability of at least one correct solution in k attempts. It suits cases where the user can retry or the system picks the best result itself.
For a feature that answers only once, one correct answer in 10 runs is enough for pass@10, but the user still receives 9 wrong answers. That is why the example in step 4 counts passes instead of using pass@k.
Step 5: when the property is meaning, not structure
Some properties can’t be counted, such as “the paraphrase must keep the original meaning”. The tianpan.co article suggests comparing embedding similarity and asserting a score of 0.8 or higher.
The 0.8 figure is only a rule of thumb, and you need to calibrate it on your own data. The code below is pseudocode. similarity is a function you write with whatever embedding model you use.
# Pseudocode: similarity() is your own implementation
def test_paraphrase_keeps_meaning():
original = "2-bedroom apartment near a school"
rewritten = rewrite(original)
assert similarity(original, rewritten) >= 0.8
The same article uses Python’s Hypothesis library to generate inputs automatically, which surfaces bugs that hand-written evals miss. Do this step last. Until steps 2 and 4 are stable, generating more inputs only gives you more failing tests.
Common mistakes
The first mistake is putting everything into one giant test. When it fails, you can’t tell which scenario broke. Follow Husain’s advice and write one test per scenario.
The second is letting a JSON exception crash the whole loop in step 4. A run where the model returns plain text should count as one failure. It shouldn’t stop the whole test.
The third is harder to spot: calling in an LLM grader too early. Anthropic advises designing questions so that code can grade as many of them as possible. Checking JSON and counting elements is cheap and fast, and gives the same result every time.
Where this skill shows up in client work
Picture the second week of an engagement. The client asks, “Is the bot OK?”, and a vague answer costs you their trust.
With the tests above, you can be specific: the address-search scenario passes 7 runs out of 10. Does the team accept that, or does it need 9/10?
That question turns a hunch into a product decision with a named owner.
On your CV, don’t write “wrote tests for AI features”. Say that you replaced string-matching tests with property assertions, set a pass-rate threshold with the client, and show how that rate changed after a prompt fix.
A small repo that covers the five steps is evidence you can easily bring to an interview.
When you look at a failing test again, don’t start by asking what the model got wrong. First check whether the test is asking the model for something the user doesn’t actually need.
Was this article useful?
Thanks for the feedback!