Test AI integrations from the cheapest layer up
Assertions and contracts are cheap, run fast and catch errors early; evals and end-to-end tests can wait, and there should be fewer of them.
In brief
- For agents, each layer of the pyramid reflects how much uncertainty you can accept, as well as a type of test.
- Start with assertions and contracts: they are cheap, run fast and catch integration failures with the client's systems.
- Begin evals with 20-50 real failure cases, and keep end-to-end tests to the few most critical flows.
An agent that gets an invoice right 90% of the time will get it right three runs in a row only 72.9% of the time. That gap decides whether it is ready to post entries to a client’s books, and it is one reason the order in which you build tests matters.
At a client site, there is rarely time to build every layer of tests for an LLM integration in the first week. You have to choose what to write first. Choose badly and you end up with a suite that is slow, flaky and still misses the failures the client will spot before you do.
The test pyramid, as Ham Vocke sets it out in “The Practical Test Pyramid” on Martin Fowler’s blog, has one main rule: the higher the layer, the fewer the tests. When you put a language model inside a client’s system, that rule still holds. What changes is what each layer means.
Block’s engineering team describes the shift neatly. With agents, the layers of the pyramid are no longer just different kinds of test. They tell you how much uncertainty you are willing to accept.
Four layers, ordered by uncertainty
The bottom layer is deterministic assertions. Hamel Husain, who writes widely on evals for AI products, calls this Level 1 and insists it must never be skipped. Assertions check what code can check: does the JSON parse, are all the fields present, has any data leaked that should not have.
The second layer is contracts. Vocke’s piece on Martin Fowler’s blog relies on consumer-driven contracts, in which the caller of an API decides what the contract must contain. The Pact tool calls the contract between consumer and provider a “pact”.
During verification, Pact compares the provider’s real responses with the minimal responses recorded by the consumer-side tests. That way the tests check only what you actually use.
Block carried this idea over to agents. They record real interactions with tools once, replay that recording on every later run, and assert on the sequence of tool calls.
The model may still phrase things differently from run to run, but the order of calls to external systems stays fixed. The example below goes one step further than Block and checks the call arguments as well.
The third layer is evals. Anthropic defines an eval simply: give an AI system an input, then apply grading logic to its output. For agents, that grading logic usually combines three kinds: code-based grading, model-based grading and human grading. This layer is therefore no longer fully deterministic.
At the top sit end-to-end tests. The piece on Martin Fowler’s blog warns that they are notoriously flaky, failing for reasons nobody can predict. Put a non-deterministic model in the middle and the flakiness only gets worse.
| Layer | What it checks | When it fails, the usual cause |
|---|---|---|
| Assertion | Schema, required fields, hard rules | A code bug, or a prompt that breaks the format |
| Contract / replay | The client’s API and the sequence of tool calls | The client changes its API, the agent calls in the wrong order |
| Eval | Output quality on real cases | A prompt change, a model change, new data |
| End-to-end | The whole flow from user to target system | Network, environment, latency, everything above |
Walking through a real integration
Take a hypothetical case. You build an agent that reads suppliers’ PDF invoices, extracts the amount and the purchase order number, then calls the client’s ERP API to create a journal entry. Failures can come from three directions: the model misreads, your code mishandles, or the ERP API returns something you did not anticipate.
The assertion layer takes an afternoon to write and runs in under a second:
def test_extraction_shape(extract):
out = extract("fixtures/invoice_017.pdf")
assert set(out) >= {"vendor_id", "amount", "currency", "po_number"}
assert out["amount"] > 0
assert out["currency"] in {"USD", "EUR", "VND"}
assert out["po_number"].startswith("PO-")
This test does not care how the model phrases anything. It only blocks outputs the downstream code certainly cannot handle.
The contract layer is the most valuable one for an FDE, because the client’s API is the thing you do not control. You record one real call to the ERP in staging, save it to a file, then write a replay test:
def test_posting_flow(replay):
calls = replay.run("cassettes/invoice_017.json")
assert [c.tool for c in calls] == ["lookup_po", "create_journal_entry"]
assert calls[1].args["amount"] == calls[0].result["open_amount"]
If the agent skips the purchase order lookup and posts the entry directly, the replay test goes red. But a cassette is only a recording of an old call. If the client’s IT team renames the open_amount field next week, the replay stays green, because it never touches the real API.
That failure needs a separate provider-side check, in the style of Pact’s verification: take the minimal responses recorded by the consumer-side tests and compare them with the real responses from the staging API. Replay keeps the agent calling in the right order; provider-side verification tells you when the client’s API changes. You need both.
For evals, Anthropic advises starting small: 20-50 simple tasks drawn from real failures are a good enough start. A two-page invoice, an invoice with the amount written in European format, an invoice missing its purchase order number: all belong in the first set. Pull them from logs and tickets rather than inventing them at your desk.
Why 90% can mislead
Anthropic distinguishes two measures. pass@k is the probability that at least one of k attempts succeeds. pass^k is the probability that all k attempts succeed, and it is the real measure of reliability.
Run the numbers for an invoice case the agent gets right 90% of the time on each run. pass@3 is 1 − 0.1³, or 99.9%, which looks excellent. pass^3 is 0.9³, or 72.9%. If the agent posts to the client’s books automatically, the second figure is the one you must report.
It need not be 100%. Husain argues that the pass rate is a product decision, depending on which failures you can live with. A flow with a human reviewer downstream might accept 72.9%. An automated payments flow would not.
This is also why Block redefines regression. In a non-deterministic system, a change in output is not yet a regression. A regression is a fall in the success rate. When you change a prompt, compare pass^k before and after instead of diffing each answer.
What to build in the first two weeks
Husain ranks the layers by cost: Level 3 is more expensive than Level 2, and Level 2 more expensive than Level 1. That argument is enough to settle the order. On day one, write assertions for every output the downstream code uses.
Next, as soon as you have access to the client’s staging environment, record real tool calls and set up the replay tests and provider-side verification. Do this early: you can only record cassettes while you still have staging access, and once you have them, most later tests run without touching the client’s systems.
In the second week, gather the first 20 failure cases into an eval set, run each case several times and record pass^k. End-to-end tests come last: just a few flows whose failure the client would certainly notice, run on a schedule rather than blocking every merge.
Common mistakes
The most common mistake is an inverted pyramid. You have dozens of end-to-end tests calling the real model and the real ERP, each run takes ten minutes and fails at random. After a few weeks the team is used to red and no longer trusts the suite.
The second is handing all grading to another model when many criteria can be checked in code. Code-based grading is cheap and deterministic, so use it wherever possible. Save model-based grading for what code cannot judge.
The third is setting a 100% target for evals and getting stuck there, or the reverse, treating every change in wording as a regression. Both have you chasing random fluctuations while the agent’s real quality does not budge.
As a career skill, this one is easy to show. On a CV, instead of writing “wrote tests for AI systems”, say that you built a contract layer against the client’s API and tracked pass^k across every prompt change.
In an interview, if asked how you know an agent is ready for production, draw this pyramid and explain why you build the lower layers first.
On the day an agent goes to production, the client usually cares about one thing: the next time the model changes, how will they know it has not broken their books? The contract layer and the pass^k figure are an answer you can back with data.
Was this article useful?
Thanks for the feedback!
5 sources
- The Practical Test Pyramid · 2018-02-26
- How Pact works
- Your AI Product Needs Evals · 2024-03-29
- Demystifying evals for AI agents · 2026-01-09
- Testing Pyramid for AI Agents · 2026-01-12