FDE Times dataFDE jobs open 924▲33 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 80Companies hiring 196Remote-friendly 25%Median US pay $200kTop hirer Databricks 107
中文VI

The newspaper of the Forward Deployed Engineer

Guides

In an AI bake-off, whoever builds the eval sets the rules

Prompts tuned on a curated demo account break in production; a held-out test on real customer data shows where you lose before the contract is signed.

ShareLinkedInFacebookX

In brief

  • Demo data is curated; production data is messy. Winning on the demo says nothing about the first month in production.
  • LLM judges are biased by position, by length and towards models from their own maker. Score each pair twice with the order swapped, use a neutral model and calibrate the judge against human labels.
  • The customer holds the held-out set. You tune only on the dev set, and that is your proof that you did not tune for the test.
Two stacked bars on the same 0–200 held-out email scale. Judged one way with A first, A wins 130. Judged both ways, with a tie whenever the verdict flips: A 85, tie 55, B 60. The 55 tie segment is highlighted in orange.
Judged one way with A first, A wins 130 of 200 cases. Judged in both orders, with a tie whenever the verdict flips, the honest result is 85–60–55. Source: hypothetical bake-off example in the article (complaint emails at a logistics company).

The customer’s email is short: in two weeks they will run the same set of questions through your product and a competitor’s, then pick one.

Many FDEs react by opening the demo account and tweaking prompts until every answer looks good. That may win the bake-off, but the system will then break in its first month in production.

There is a better way. Offer to build the test yourself, and keep it fair enough that the customer trusts the result. Whoever holds the ruler defines what “good” means. If the ruler is straight, you gain an edge and you also learn exactly where you lose before the contract is signed.

The skill applies to any AI vendor selection, even when the competitor is another model from your own company.

Why does winning on the demo tell you nothing?

A practitioner’s post on the tianpan.co blog identifies the core gap: demo accounts hold curated, balanced data, while production accounts hold messy data with missing fields, and are often nearly empty.

Tuning prompts on the demo means optimising for exactly the data the customer will never see again. The post also argues that the FDE is usually the person who should own this eval.

Anthropic’s guidance on building evals sets out the first principle: an eval must reflect the real distribution of tasks. Hamel Husain, who writes and teaches about evals, puts it more bluntly.

In his view, generic off-the-shelf metrics waste time and create false confidence when used as a measure of quality. Differences between models must be measured on your own data, which here means the customer’s data.

So the first job when you sit down with the customer is not writing prompts. You ask for a sample of real logs from the system they run today, including the hard cases. Anthropic is explicit about not forgetting edge cases, including ambiguous ones on which humans themselves disagree.

The ruler can be bent too

Nobody can hand-score a bake-off with several hundred questions, and nobody should. Anthropic advises favouring volume: many automatically graded questions with a slightly weaker signal beat a handful of carefully hand-graded ones. But automated grading means using an LLM as a judge, and judges have biases of their own.

The paper “Judging LLM-as-a-Judge” by Zheng et al. names three: position bias, where the judge favours answers by the order they appear in; verbosity bias, where it favours longer answers; and self-enhancement bias, where it favours its own output.

OpenAI’s guidance also lists the first two as known challenges of pairwise grading. The third is the most dangerous in a bake-off: if the judge is a model from one of the two vendors, the result is tilted from the start. Anthropic’s code examples note that it is generally best to grade with a different model from the one that produced the output.

The good news is that strong judges remain usable. The same paper finds that strong judges agree with human graders in over 80% of cases, on a par with how often two humans agree with each other.

The condition is that you verify this on the customer’s problem rather than assume it. Hamel Husain advises testing the judge on human-labelled examples of exactly the kinds of error it needs to catch.

A worked bake-off

Imagine the customer is a logistics company that wants AI to draft replies to complaint emails. You take 300 real emails from the previous quarter. Thirty of them are deliberately hard: empty emails, malformed tracking numbers, customers writing in a mix of two languages, vague requests.

You split the set into 100 dev emails and 200 held-out emails. Both sides may tune prompts freely on the dev set. The customer keeps the held-out set and runs it only once, on the day of the test. Lyft’s engineers, presenting at AI Engineer World’s Fair 2026, used exactly this dev/held-out split to detect overfitting.

Now the scoring. Each of the 200 held-out emails is judged twice, once with A’s answer first and once with B’s first, for 400 judgements in all. Suppose that with A first, A wins 130/200. With B first, A wins only 95/200. That gap of 35 cases is position bias, in plain view.

The scoring rule is simple: a side counts as winning only if it wins in both orders; anything else is a tie. Suppose A wins both rounds in 85 cases, B wins both in 60, and 55 flip and count as ties.

The honest result is 85–60–55, not “A wins 65%”, which is what you would report by running one way only with A first.

def judge_pair(email, ans_a, ans_b, judge):
    # judge is a third-party model, not from either vendor in the contest
    first = judge(email, left=ans_a, right=ans_b)   # returns 'left' or 'right'
    second = judge(email, left=ans_b, right=ans_a)
    a_wins_first = first == "left"
    a_wins_second = second == "right"
    if a_wins_first and a_wins_second:
        return "A"
    if not a_wins_first and not a_wins_second:
        return "B"
    return "tie"  # verdict flipped when the order was swapped

Two more layers of checks remain. OpenAI’s guidance notes that models sometimes give different outputs for the same input. So run each system three times on every email and report the variance as well, not just the best run.

OpenAI also advises controlling for length. Compare the average answer length of the winning side; if it is markedly longer, rewrite the rubric so the judge scores correct content, not fullness.

Finally, calibrate the judge. Ask the logistics company’s customer service team to label 50 pairs, then compare their labels with the judge’s verdicts. OpenAI describes this step as using human feedback to calibrate automated grading. Lyft’s engineers warn that agreement with an unvalidated judge does not prove the agent is working correctly.

Steps at the customer

Start with a one-page set of rules, sent to the customer before anyone writes a line of prompt. It states the data source, the share of edge cases, how dev and held-out are split, the judge model and why it was chosen, the tie rule, and the number of repeat runs.

Invite the competitor to read it too, if the customer agrees. Rules you are willing to show a competitor are rules the customer will trust.

Next, gather the real data and deliberately mix in hard cases. Then write the rubric with the customer’s domain staff, calibrate the judge on their labels, and run the held-out set only once the judge reaches acceptable agreement. The final report should give three numbers, wins, losses and ties, plus a few concrete cases you lost.

That last detail is the one most often skipped. Pointing out where you lost is the fastest way to make the customer believe where you won, and it doubles as the task list for the first week of deployment.

Mistakes that make a bake-off worthless

The most common is peeking at the held-out set. Change a prompt after seeing a few held-out questions and that set has become a dev set; you have lost your only evidence that you did not tune for the test. The second is using a model from the same maker as one of the contestants as judge, simply because it is convenient and cheap.

The third is running one way, once, then reporting a win rate to one decimal place. The figure looks precise but can reverse just by swapping the order. The fourth is using a public benchmark instead of the customer’s data, exactly the false confidence Hamel Husain warns about.

This skill also belongs on a job application. On a CV, a line such as “built a bake-off eval of 300 cases from production logs, with a judge calibrated against human labels; the customer used the results to make its decision” carries far more weight than “experienced with LLMs”.

In interviews, be ready to explain why you count a flipped verdict as a tie.

Every bake-off ends with a purchasing decision. But the eval you leave behind stays in use long afterwards, and it will protect you every time the model is upgraded.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
6 sources
Read next on the roadmap · Stage 6: MeasurementAn agent that says it cancelled the order may not haveA τ-bench-style grader checks the database rather than the transcript, and takes an afternoon of Python on a laptop to build.