# The best AI agent fully solves just 14 of 95 customer handoffs

> FDE-Gym scores agents on the customer's own success metric, and the thing that trips them up most is the constraint nobody says out loud.

Original: https://fdetimes.net/en/news/fde-gym-benchmark-ai-agents-customer-delivery/

Given 95 tasks that involve handing a working system over to a customer, the strongest AI agent configuration fully solved only 14. That is the headline result of FDE-Gym, a benchmark posted to arXiv on October 15, 2026.

The authors tested 13 model configurations from seven providers. In the most realistic setting, the best of them averaged a score of just 0.56.

What is new is the yardstick. FDE-Gym does not check whether the code runs or the tests pass. It scores an agent on how much the delivered system improves **the customer's own success metric**, which is close to how a real customer judges the outcome.

For anyone working as an FDE, or hoping to become one, the result matters because it measures the last mile: turning what an agent builds into results inside the customer's environment. On that stretch, even the strongest configuration left most tasks unsolved.

The benchmark measures agents, not people, so what follows is an inference rather than a conclusion: that last mile is probably where FDEs can still make a difference, and the skill it demands can be trained.

## Where the machines stumble

The authors argue that real customer deployments differ in kind from benchmark tasks with clearly specified problems. They identify the main bottleneck as finding the constraints nobody puts into words.

Take a hypothetical example. A customer wants to cut the time it takes to handle support tickets, and the agent builds a neat automatic triage pipeline.

Only in the second week does it emerge that tickets from enterprise accounts must be read by a person first, because of a contract clause. The code is still correct, yet the customer's metric does not budge. That is precisely the kind of failure FDE-Gym's yardstick is designed to catch.

The way the tasks are built targets the same weakness. Experts took 20 real deployment projects across 18 domains, turned them into customer operating environments, and then **changed a few key facts** to create variants.

They kept only tasks whose graders had been verified. Because the key facts differ, a familiar solution rarely carries over intact: to pass, an agent has to reread the context of each variant.

**Key point:** Working code is only half the job; the customer cares only whether their metric moves.

## Narrow is easy, end to end is hard

Set against recent benchmarks, the boundary is fairly clear. FDE-Bench, by Ding, Zhan, Li and Guo and published on September 23, 2026, measures only a narrow slice of FDE work: configuring deployment environments with Docker, Compose and Kubernetes. On its 136 tasks, seven models solved between 52.9% and 75.0%.

The two benchmarks differ in their tasks and scoring, so they cannot be compared directly. Even so, the trend is plain: the narrower and better specified the task, the better agents perform. When they have to work out for themselves what the customer needs, scores fall sharply.

Another study by Ding and Zhan, "Trains but Doesn't Learn" (September 21, 2026), treats delivering a post-trained model as FDE work. The agent goes through ten stages, each scored from facts recorded on the platform.

Its central failure mode will sound familiar: a run looks entirely fine, but the final model is no better than the base model.

## Which skills are worth more

An open-source project with the same name on GitHub, deeplethe/fde-gym from DeepLethe, builds a full practice platform around the same idea: each exercise is drawn from a real customer handoff and done alongside a coding agent.

The project's argument is that once AI writes correct code, correct code no longer separates strong engineers from ordinary ones. What still separates them is the ability to produce results when the system is running for real at the customer's site.

So if you are preparing for an FDE role, do not spend all your time drilling algorithms. Practice customer discovery: in the first meeting, ask how the customer measures success, who has the authority to say "no," and which data must never leave the system. On your CV, replace lines like "built X" with how the customer's metric changed.

When reading job descriptions or interviewing, ask directly: is this role accountable for the customer's outcome, or only for the code it is assigned? The figure of 14 out of 95 says nothing about people, since the benchmark scores only agents. But it shows that the last mile of deployment is where machines are still weakest, and that is where your effort is best spent.

**Try this week:**

- Pick your most recent project, list every constraint you only discovered after you had started building, then write the question that would have surfaced each one in the first meeting
- Rewrite one line on your CV from 'built X' to 'improved the customer's metric Y from A to B', using your real numbers
- Before your next discovery session, write down the customer's success metric in their own words and ask them to confirm it

## Sources

- [Paper page - FDE-Gym: The Last Mile of AI Delivery](https://huggingface.co/papers/2610.22350)

- [FDE-Gym: The Last Mile of AI Delivery](https://arxiv.org/abs/2610.22350)

- [FDE-Gym: The Last Mile of AI Delivery (HTML version)](https://arxiv.org/html/2610.22350)

- [FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration](https://arxiv.org/abs/2609.27571)

- [Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers](https://arxiv.org/abs/2609.25237)

- [deeplethe/fde-gym (GitHub)](https://github.com/deeplethe/fde-gym)
