# Test AI integrations from the cheapest layer up

> Assertions and contracts are cheap, run fast and catch errors early; evals and end-to-end tests can wait, and there should be fewer of them.

Original: https://fdetimes.net/en/guides/ai-integration-testing-pyramid/

An agent that gets an invoice right 90% of the time will get it right three runs in a row only 72.9% of the time. That gap decides whether it is ready to post entries to a client's books, and it is one reason the order in which you build tests matters.

At a client site, there is rarely time to build every layer of tests for an LLM integration in the first week. You have to choose what to write first. Choose badly and you end up with a suite that is slow, flaky and still misses the failures the client will spot before you do.

The test pyramid, as Ham Vocke sets it out in "The Practical Test Pyramid" on Martin Fowler's blog, has one main rule: the higher the layer, the fewer the tests. When you put a language model inside a client's system, that rule still holds. What changes is what each layer means.

Block's engineering team describes the shift neatly. With agents, the layers of the pyramid are no longer just different kinds of test. They tell you how much uncertainty you are willing to accept.

## Four layers, ordered by uncertainty

The bottom layer is deterministic assertions. Hamel Husain, who writes widely on evals for AI products, calls this Level 1 and insists it must never be skipped. Assertions check what code can check: does the JSON parse, are all the fields present, has any data leaked that should not have.

The second layer is contracts. Vocke's piece on Martin Fowler's blog relies on consumer-driven contracts, in which the caller of an API decides what the contract must contain. The Pact tool calls the contract between consumer and provider a "pact".

During verification, Pact compares the provider's real responses with the minimal responses recorded by the consumer-side tests. That way the tests check only what you actually use.

Block carried this idea over to agents. They record real interactions with tools once, replay that recording on every later run, and assert on the sequence of tool calls.

The model may still phrase things differently from run to run, but the order of calls to external systems stays fixed. The example below goes one step further than Block and checks the call arguments as well.

The third layer is evals. Anthropic defines an eval simply: give an AI system an input, then apply grading logic to its output. For agents, that grading logic usually combines three kinds: code-based grading, model-based grading and human grading. This layer is therefore no longer fully deterministic.

At the top sit end-to-end tests. The piece on Martin Fowler's blog warns that they are notoriously flaky, failing for reasons nobody can predict. Put a non-deterministic model in the middle and the flakiness only gets worse.

| Layer | What it checks | When it fails, the usual cause |
|---|---|---|
| Assertion | Schema, required fields, hard rules | A code bug, or a prompt that breaks the format |
| Contract / replay | The client's API and the sequence of tool calls | The client changes its API, the agent calls in the wrong order |
| Eval | Output quality on real cases | A prompt change, a model change, new data |
| End-to-end | The whole flow from user to target system | Network, environment, latency, everything above |

**Key point:** In an AI system, each testing layer is a level of uncertainty you can accept.

## Walking through a real integration

Take a hypothetical case. You build an agent that reads suppliers' PDF invoices, extracts the amount and the purchase order number, then calls the client's ERP API to create a journal entry. Failures can come from three directions: the model misreads, your code mishandles, or the ERP API returns something you did not anticipate.

The assertion layer takes an afternoon to write and runs in under a second:

```python
def test_extraction_shape(extract):
    out = extract("fixtures/invoice_017.pdf")
    assert set(out) >= {"vendor_id", "amount", "currency", "po_number"}
    assert out["amount"] > 0
    assert out["currency"] in {"USD", "EUR", "VND"}
    assert out["po_number"].startswith("PO-")
```

This test does not care how the model phrases anything. It only blocks outputs the downstream code certainly cannot handle.

The contract layer is the most valuable one for an FDE, because the client's API is the thing you do not control. You record one real call to the ERP in staging, save it to a file, then write a replay test:

```python
def test_posting_flow(replay):
    calls = replay.run("cassettes/invoice_017.json")
    assert [c.tool for c in calls] == ["lookup_po", "create_journal_entry"]
    assert calls[1].args["amount"] == calls[0].result["open_amount"]
```

If the agent skips the purchase order lookup and posts the entry directly, the replay test goes red. But a cassette is only a recording of an old call. If the client's IT team renames the `open_amount` field next week, the replay stays green, because it never touches the real API.

That failure needs a separate provider-side check, in the style of Pact's verification: take the minimal responses recorded by the consumer-side tests and compare them with the real responses from the staging API. Replay keeps the agent calling in the right order; provider-side verification tells you when the client's API changes. You need both.

For evals, Anthropic advises starting small: 20-50 simple tasks drawn from real failures are a good enough start. A two-page invoice, an invoice with the amount written in European format, an invoice missing its purchase order number: all belong in the first set. Pull them from logs and tickets rather than inventing them at your desk.

## Why 90% can mislead

Anthropic distinguishes two measures. pass@k is the probability that at least one of k attempts succeeds. pass^k is the probability that all k attempts succeed, and it is the real measure of reliability.

Run the numbers for an invoice case the agent gets right 90% of the time on each run. pass@3 is 1 − 0.1³, or 99.9%, which looks excellent. pass^3 is 0.9³, or 72.9%. If the agent posts to the client's books automatically, the second figure is the one you must report.

It need not be 100%. Husain argues that the pass rate is a product decision, depending on which failures you can live with. A flow with a human reviewer downstream might accept 72.9%. An automated payments flow would not.

This is also why Block redefines regression. In a non-deterministic system, a change in output is not yet a regression. A regression is a fall in the success rate. When you change a prompt, compare pass^k before and after instead of diffing each answer.

## What to build in the first two weeks

Husain ranks the layers by cost: Level 3 is more expensive than Level 2, and Level 2 more expensive than Level 1. That argument is enough to settle the order. On day one, write assertions for every output the downstream code uses.

Next, as soon as you have access to the client's staging environment, record real tool calls and set up the replay tests and provider-side verification. Do this early: you can only record cassettes while you still have staging access, and once you have them, most later tests run without touching the client's systems.

In the second week, gather the first 20 failure cases into an eval set, run each case several times and record pass^k. End-to-end tests come last: just a few flows whose failure the client would certainly notice, run on a schedule rather than blocking every merge.

## Common mistakes

The most common mistake is an inverted pyramid. You have dozens of end-to-end tests calling the real model and the real ERP, each run takes ten minutes and fails at random. After a few weeks the team is used to red and no longer trusts the suite.

The second is handing all grading to another model when many criteria can be checked in code. Code-based grading is cheap and deterministic, so use it wherever possible. Save model-based grading for what code cannot judge.

The third is setting a 100% target for evals and getting stuck there, or the reverse, treating every change in wording as a regression. Both have you chasing random fluctuations while the agent's real quality does not budge.

As a career skill, this one is easy to show. On a CV, instead of writing "wrote tests for AI systems", say that you built a contract layer against the client's API and tracked pass^k across every prompt change.

In an interview, if asked how you know an agent is ready for production, draw this pyramid and explain why you build the lower layers first.

On the day an agent goes to production, the client usually cares about one thing: the next time the model changes, how will they know it has not broken their books? The contract layer and the pass^k figure are an answer you can back with data.

**Try this week:**

- Take an LLM integration you are working on and write 5 assertions for its output: correct schema, all required fields present, values within valid ranges.
- Record one real tool call to the client's API, then write a replay test that checks the order of tool calls.
- Gather 20 real failure cases from logs or tickets into a first eval set, run each case 3 times and calculate pass^3.

## Sources

- [The Practical Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html)

- [How Pact works](https://docs.pact.io/getting_started/how_pact_works)

- [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/)

- [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)

- [Testing Pyramid for AI Agents](https://engineering.block.xyz/blog/testing-pyramid-for-ai-agents)
