# Customers never hand you a metric, only a complaint

> The FDE's job is to turn "the bot's answers are bad" into pass-or-fail criteria and an eval suite both sides sign off on.

Original: https://fdetimes.net/en/guides/turn-customer-complaints-into-evals/

The first meeting with a customer often ends with something like: "The bot's answers are bad. Make it better." No data, no concrete examples, and nobody says how good "better" has to be. Yet two weeks later, that same person will decide whether the project succeeded or failed.

This is where many good engineers get stuck. You can write strong prompts and build a clean RAG pipeline, but if you do not know what "good" means, you are optimising for a target nobody can see. The skill to learn here is turning a customer's complaint into criteria that a model's output either meets or fails.

## Why is this the FDE's job and not someone else's?

PostHog describes FDEs as engineers embedded in the customer's own team. That position gives you something the research team at headquarters lacks: real data, real users, real complaints. That is why companies put the job of defining success squarely in this role.

Cursor's FDE job posting asks the engineer to lead discovery with the customer's engineers to find the real bottleneck, define clear success metrics from the outset, and then own the solution end to end. Owning it end to end means you cannot diagnose and hand off; you have to prove you have fixed it.

That proof, according to a skills guide for AI FDEs on SecondTalent, is an evaluation harness showing that the system meets the definition of "good output" both sides agreed on.

The OpenAI example PostHog describes shows evals going further still: an FDE at a customer automating its call centre built evals to verify performance, then took that data back to the research team to drive further improvements. The eval is both evidence for the customer and a signal for the product.

## Start from failures, not from metrics

Many people's first instinct is to pick a popular metric, such as a "helpfulness" score or text similarity, and run it. Hamel Husain and Shreya Shankar warn that generic evals waste time and create false confidence when used as a measure of quality. They recommend building custom evaluators grounded in error analysis.

Error analysis, the two authors argue, is the most important activity in evals, because it is what decides which evals to write.

Anthropic's documentation adds a principle: criteria must be specific. Instead of "good performance", say "accurate sentiment classification". Criteria must also be tied to the application's purpose: citation accuracy may be critical for a medical application but matter far less for a casual chatbot.

## A worked example, end to end

Imagine you are an FDE at a retail company. The customer has deployed a customer-service bot and complains that "the bot's answers are bad". The first step is not opening the code. You follow up: "Could you show me the three recent conversations you thought were worst?"

Suppose those three conversations reveal entirely different problems: the bot misstated the returns policy, the bot promised a refund it had no authority to give, and the bot answered correctly but at tedious length. "Bad" is really three failures, with three different levels of severity.

Now you ask for a few dozen more logs, read them all, and write a one-line description of the failure for every bad output.

After grouping them, suppose the unauthorised refund promises turn out to be rare but the most dangerous legally, while policy errors are the most frequent. You take this table back to the customer. The conversation changes character entirely: it is no longer "make it better" but "which failure do you want to prioritise, and what threshold will you accept?"

Next come the criteria. Husain and Shankar favour binary pass/fail judgements over a 1-5 scale, because they force the person writing them to think more clearly and produce more consistent labels. For this example, the criteria might look like this:

| Complaint | Specific failure | Pass/fail criterion |
|---|---|---|
| "The bot gives wrong answers" | Misstates the returns policy | Pass if every return condition stated matches the current policy document |
| "The bot is dangerous" | Promises refunds without authority | Fail if the output commits to a refund without handing off to a staff member |
| "The bot rambles" | Answers are too long | Pass if the answer resolves the question within the agreed length limit |

Finally, the thresholds. Anthropic gives an example of turning the goal "safe outputs" into: fewer than 0.1% of outputs out of 10,000 trials flagged as toxic by a content filter. That structure, a specific task, a threshold and a number of trials, is exactly what you need to put in front of the customer for sign-off.

For the refund promises, you might propose a threshold close to zero; on length, the customer may accept something much looser.

## Do not trust a judge until you have tested it

When output volumes grow, you will use an LLM as a judge to score automatically. This is where it is easiest to fool yourself. Husain and Shankar stress that any evaluator making judgements must be tested on human-labelled examples of exactly the failure it is meant to detect.

In the example above, you would sit with one of the customer's service agents, label a set of outputs pass/fail, and compare those labels with the judge's results. If the judge misses many of the refund promises that a human catches, the "pass" figure on your dashboard means nothing.

This comparison is also a good way to earn the customer's trust in the eval, because their own people helped define what "correct" means. A worthwhile exercise: label 20 outputs yourself against one criterion, have an LLM judge score the same 20, and count where the two disagree.

Each disagreement is a question you must be able to answer before you give the customer any number.

**Key point:** A metric is only worth something when it is tied to a real failure, has a threshold the customer has agreed to, and has a judge that has been checked against real people.

## Common mistakes

The most common mistake is taking the complaint and going straight to fixing the prompt. You may fix exactly what you guessed, while what actually hurts the customer lies elsewhere, and you have no way to prove progress.

The second is using a 1-5 scale because it looks sophisticated. When two people scoring the same output give it a 3 and a 4, the average helps nobody make a decision. The third is treating evals as a one-off: when real data changes, the failure categories change, and you have to read the traces again.

The last is subtler: defining "good" yourself without having the customer confirm it. An eval written by the FDE alone is the FDE's eval. An eval the customer signs is a contract.

## Showing this skill when job hunting

When reading FDE job descriptions, look for phrases such as "define success metrics", "build evals" and "discovery". They signal that the company needs exactly this skill. On your CV, instead of writing "improved chatbot quality", write how many traces you grouped failures from, which criteria you defined, and how the pass rate changed.

In interviews, if asked "the customer says the product isn't good; what do you do?", do not rush to a technical answer. Walk through the process: ask for examples, read the traces, group the failures, write binary criteria, set thresholds with the customer, test the judge.

Being able to describe the full loop is the clearest way to show you know how to turn a complaint into a measure, not just how to fix things. Plenty of people can fix things; the person who gets a customer to nod at a number is the one handed the next project.

**Try this week:**

- Take the 30 most recent outputs from an AI feature you are working on, read each one, and write a one-line description of the failure for every bad output
- Group your notes into 3-5 failure categories, pick the largest, and write a pass/fail criterion for it
- Rewrite one vague goal in your team's backlog using the template: specific task, threshold, number of trials

## Sources

- [Forward Deployed Engineer (Cursor, Accel job board)](https://jobs.accel.com/companies/cursor-2-379534b5-1a2a-49fd-86d0-ccd3e256c4f0/jobs/83110824-forward-deployed-engineer)

- [WTF is a forward deployed engineer? (and why everyone is hiring them)](https://posthog.com/blog/forward-deployed-engineer)

- [AI Forward Deployed Engineer: Key Skills & Responsibilities in 2026](https://www.secondtalent.com/occupations/ai-forward-deployed-engineer/)

- [AI Evals: Everything You Need to Know](https://hamel.dev/blog/posts/evals-faq/)

- [Define success criteria and build evaluations](https://platform.claude.com/docs/en/docs/test-and-evaluate/define-success)