# Your clean run at the client's site is a test nobody can rerun

> Google's SRE book treats the untested as broken; a real handover leaves tests, measurable acceptance criteria and a runbook the client's team can trust.

Original: https://fdetimes.net/en/analysis/client-site-run-not-handover-proof/

The chapter on testing for reliability in Google's *Site Reliability Engineering* book opens with a short epigraph: if you haven't tried it, assume it's broken. A few paragraphs later the book goes further. Passing tests do not necessarily prove that a system is reliable, but failing tests usually prove that it is not.

Remember those two lines before you message a client to say "done, it's running on site". Look closely and a smooth run on the client's live system is just one passing test. Nobody wrote that test down and nobody can run it again.

This matters because every FDE engagement ends in a handover. Educative's course on FDEs for AI systems says a client's team cannot operate what it does not understand, and cannot maintain what was designed only for the FDE. From handover day, the question changes from "does it work?" to "is it safe to change?"

## What does one smooth run prove?

Take a typical case. You write a script for a client that reads order export files from a legacy system, normalises dates and currency units, and loads the data into a new warehouse. It runs cleanly on the last three days of data. The client looks at the dashboard and nods.

That run proves three things: the file format for those three days matched your assumptions, the access permissions were correct, and the pipeline did not fail at any step.

It says nothing about what happens when a branch exports dates in a different format, when a row is missing its customer ID, or when one of the client's engineers renames a column next month. None of those cases has happened yet, so the smooth run cannot tell you anything about them.

To be fair, running on real data is essential. PostHog's explainer on the FDE role describes FDEs working with clients until the client sees real results on real data. But that is the condition for sign-off. It is not evidence that the system can survive change.

The SRE book identifies the difference. Every test that passes both before and after a change reduces the uncertainty an analyst has to allow for. The key words are "before and after". A smooth run on site has no "after", because nobody runs it again when the code changes.

## Why does FDE code so often lack tests?

The way the work is organised explains most of it. When you are writing for a single client and need results before the demo, tests are the easiest thing to cut.

The reasoning sounds sensible: the code is used in one place, and you are right there to fix it if it breaks. That reasoning holds only until the day you leave.

A Substack piece on the Field-to-Product model, from Second File, warns about two things: custom code that lives only in one client's account, and "hero" engineers who become single points of failure. Untested code turns you into exactly that kind of hero, because only you know what it is and is not supposed to do.

## The people who inherit the code pay the price

*Software Engineering at Google* says that an equally important reason to test software is to keep the ability to change it. For a project about to be handed over, the consequence is direct. Untested code can still be changed, but every change is a gamble for the client's team.

The cost also grows over time. The same book says the later a bug is found in the development lifecycle, the more expensive it is, in many cases exponentially so. For FDE code, the latest possible point is when a bug surfaces in the client's production system after its author has moved on to another project.

Tests also do something written documentation rarely does well. A short test that checks a single behaviour tells the client's team how you intended the script to handle rows with no customer ID. It says so more reliably than any wiki page, because it runs.

**Key point:** At handover, the question is not whether the code runs, but whether someone else can safely change it.

## What does handover evidence look like?

If one smooth run is not enough, what is? The table below puts the smooth run in the first row as a baseline, then adds five other kinds of evidence. Each answers a different question, and none can replace another.

| Evidence | Proves | Does not prove | Who needs it most |
|---|---|---|---|
| One smooth run at the client's site | The code did not break on that day's data and configuration | The code can handle new data, new configuration or new changes | The client, during the demo |
| A numeric definition of "working", agreed at the start | Both sides agree on what success means | Whether the code is easy to maintain | The project sponsor on the client side |
| Automated tests, one behaviour per test | Intended behaviour survives each change | The code meets the client's business goals | The client's engineers when they change the code |
| Regression tests from bugs found on site | Known bugs do not come back | Bugs nobody has hit yet | Whoever changes the code where each bug occurred |
| Scheduled production tests | The live system stays healthy over time | Whether a specific change broke a behaviour | The operations team |
| An upstream PR with a reproduction case | The fix has a place in the core product | Whether the client's remaining custom code is sound | The product team |

An FDE playbook by Truong Phung on dev.to sums up the difference with a comparison: consultants leave documents behind, while FDEs leave a running system and the organisational capability to operate it. The five kinds of evidence below the first row of the table are that capability, in a form the client's team can check for itself.

The playbook also advises writing the leave-behind deliverables, including evals, runbooks and training for the client-side owner, into the engagement definition from the start rather than leaving them until the end. Without them, the author warns, the system will rot within 90 days of your departure.

## Turn every bug on site into a test

You do not need to write a full test suite for old code. The most modest place to start is to record each bug found on site as a test case. For an FDE this costs almost nothing extra, because when you fix the bug you already have the input that caused it.

Go back to the order script. Suppose that in week two a branch sends a file with dates written day-month-year, and the script reads them as month-day-year.

You fix it, then spend ten more minutes writing a test with exactly those three rows of data, checking that the dates are read correctly. From then on, any client engineer who touches the date-parsing function will see the test fail straight away if they break that case again.

The next step is to move whatever is worth keeping into the product. The Field-to-Product piece puts the generalisation step in one line: push abstractions and tests upstream, and retire local patches.

The date-parsing bug above is a good candidate. If the product has an import feature, that test belongs in the product repository, not just in one client's custom folder.

Timing is what decides it. Most of this work has to start in the first week of the engagement, not the last, and the reason lies in what late tests actually do.

## What does a test written in the final week check?

The most common mistake is to treat a production check as enough. Production tests are necessary, but they only show that the system is still healthy. They cannot tell you which behaviour your last change broke.

A subtler mistake is to cram the tests into the final week. Tests written then tend to record what the code currently does. A freeCodeCamp tutorial is explicit that this kind of test does not tell you the existing behaviour is correct; the author writes that they are only recording the behaviour, not pretending it is right.

That technique is useful when refactoring legacy code. But if it is the whole test suite you hand over, you are freezing your own mistakes and passing them to the client's team as if they were intended behaviour. Tests written early, while you are still asking the client "what should happen to rows with no ID?", record intent rather than current behaviour.

Another trap is a single giant test that checks the whole pipeline from input file to dashboard. When it goes red, the client's engineers cannot tell which behaviour broke. The last trap is fixing a bug on site without leaving a test, and keeping the fix in a custom branch indefinitely. The bug can come back, and the core product never learns about it.

## What do hiring managers want to see?

On a CV, "deployed a pipeline for client X" is worth no more than a smooth run. "Handed over with a regression test for every bug found on site; the client's team shipped later fixes on its own" says far more.

In interviews, be ready to walk through one specific bug: how you reproduced it, what the test looked like, and which repository the test lives in now.

A smooth run on site gets you the acceptance sign-off. Whether the handover was any good only becomes clear the first time the client's team changes the code without having to call you.

**Try this week:**

- Open the most recent custom code you wrote for a client and list three behaviours the client's team will need to change. Write a short test for each, with each test checking only one behaviour.
- Pick a bug you found at a client site and turn it into a regression test that reproduces the exact input that caused it.
- Before your next engagement, write down a numeric definition of "working" and get the client to confirm it before you start on production code.

## Sources

- [Testing for Reliability (Site Reliability Engineering, Ch. 17)](https://sre.google/sre-book/testing-reliability/)

- [Testing Overview (Software Engineering at Google, Ch. 11)](https://abseil.io/resources/swe-book/html/ch11.html)

- [The Forward-Deployed Engineer Playbook (DEV Community, Truong Phung)](https://dev.to/truongpx396/the-forward-deployed-engineer-playbook-23d9)

- [Field‑to‑Product: an operating system for Forward‑Deployed Engineers](https://secondfile.substack.com/p/fieldtoproduct-an-operating-system)

- [WTF is a forward deployed engineer? (and why everyone is hiring them)](https://posthog.com/blog/forward-deployed-engineer)

- [How to Build Characterization Tests Before Refactoring Legacy Code](https://www.freecodecamp.org/news/characterization-tests-before-refactoring-legacy-code/)

- [The Project Handoff (Forward Deployed Engineer for AI Systems, Educative)](https://www.educative.io/courses/forward-deployed-engineer/the-project-handoff)