Evals are becoming the FDE's definition of done, and a contract term too

Eval pass rates are about to turn up somewhere few engineers expect: in commercial agreements with customers. That is the argument FDE Hub makes in “The Definition of Done”, published in mid-September, and it takes aim at a habit common among deployment teams: the system is running, the customer is using it, so the job is done.

Milos Mandic and John (SuccessVP), the two authors behind FDE Hub, argue that evals are becoming the definition of done for AI work, not just for writing code. For an engineer moving into a forward deployed role, this changes how you prove your value. Not with a demo that runs, but with a graded test suite.

From internal metric to contract term

The most striking part of FDE Hub’s argument is the dual role of evals: inside the company they are a metric; outside, they are a contract between vendor and customer. The pass rate is therefore no longer the engineering team’s private business but something both vendor and customer commit to.

PostHog, in the handbook for its FDE team, arrives at the same place by a different route. Before delivery begins, every engagement needs a measurable outcome, a rough timeline and a definition of done agreed by both sides. PostHog measures engagements by what changed for the customer, and settles how to measure it before the work starts.

Scope is locked to the definition of done as a result. PostHog is blunt: a deliverable that doubles in scope is a new quote, not an informal extension. When “done” is defined by evals, every additional request has something to be checked against, and the FDE no longer has to argue with the customer on the basis of each side’s impressions.

Why “running” is not enough

Anthropic defines an eval concisely: give an AI an input, then apply grading logic to the output to measure success. Teams without evals are stuck in a reactive loop, catching failures only once they reach production. That is the familiar “running” state: the system is up, but it is up so you can patch whatever the customer just reported.

Having evals is not enough either. Anthropic singles out regression evals with one question: can the agent still handle every task it could handle before? FDE Hub pushes the idea further. The moat was never the eval file; it is an eval library maintained across many customers.

An eval file nobody updates gradually ends up answering questions nobody is asking any more. FDE Hub warns about exactly this: a high pass rate on a stale eval suite can mislead, and the flattering number lulls both the team and the customer.

Start from logs, not from a rubric

So how should evals be written? Hamel Husain points to a common mistake: writing a rubric before looking at any data. The right order is to analyse real logs first, define failure from what the system actually gets wrong, and only then turn those failures into test cases.

For an FDE, the customer’s logs are the raw material, and the right to sit beside the customer and read them is an advantage engineers back at headquarters do not have.

FDEs themselves are judged by the same yardstick. FDE Academy says FDE performance is assessed on a documented evaluation harness, with graded test cases and clear definitions of failure, not on a demo that runs.

If you are preparing a CV for an FDE role, change how you describe your projects. Instead of “deployed a chatbot for Bank X”, write how many cases the eval suite contained, which logs they came from, how far the pass rate rose, and who on the customer side signed off on the definition of done.

When reading job descriptions, look for the words eval, harness and measurable outcome; if they are missing, ask in the interview who decides when an engagement is done.

The answer will tell you whether the company is selling demos or selling outcomes. PostHog has chosen to measure by what changes for the customer; FDE Academy has chosen to grade FDEs on a harness rather than a demo. If you have to place a bet, back the side that is measuring.