FDE Times dataFDE jobs open 924▲33 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 80Companies hiring 196Remote-friendly 25%Median US pay $200kTop hirer Databricks 107
中文VI

The newspaper of the Forward Deployed Engineer

News

The best AI agent fully solves just 14 of 95 customer handoffs

FDE-Gym scores agents on the customer's own success metric, and the thing that trips them up most is the constraint nobody says out loud.

ShareLinkedInFacebookX

In brief

  • FDE-Gym has 95 tasks built from 20 real deployment projects across 18 domains, scored on the customer's success metric
  • Across 13 model configurations from seven providers, the best averaged a score of just 0.56 and fully solved 14 of 95 tasks
  • The authors argue the main bottleneck for agents is uncovering the constraints customers never state, the core of FDE work
A large orange "14" next to "/95". On the right is a grid of 95 dots, 14 of them filled orange, marking the handover tasks that the strongest AI agent configuration fully solved on FDE-Gym. The other 81 dots are grey.
Of 13 model configurations tested on FDE-Gym, the strongest fully solved only 14 of 95 tasks involving the handover of systems to customers. Source: FDE-Gym, benchmark published on arXiv on 15 October 2026.

Given 95 tasks that involve handing a working system over to a customer, the strongest AI agent configuration fully solved only 14. That is the headline result of FDE-Gym, a benchmark posted to arXiv on October 15, 2026.

The authors tested 13 model configurations from seven providers. In the most realistic setting, the best of them averaged a score of just 0.56.

What is new is the yardstick. FDE-Gym does not check whether the code runs or the tests pass. It scores an agent on how much the delivered system improves the customer’s own success metric, which is close to how a real customer judges the outcome.

For anyone working as an FDE, or hoping to become one, the result matters because it measures the last mile: turning what an agent builds into results inside the customer’s environment. On that stretch, even the strongest configuration left most tasks unsolved.

The benchmark measures agents, not people, so what follows is an inference rather than a conclusion: that last mile is probably where FDEs can still make a difference, and the skill it demands can be trained.

Where the machines stumble

The authors argue that real customer deployments differ in kind from benchmark tasks with clearly specified problems. They identify the main bottleneck as finding the constraints nobody puts into words.

Take a hypothetical example. A customer wants to cut the time it takes to handle support tickets, and the agent builds a neat automatic triage pipeline.

Only in the second week does it emerge that tickets from enterprise accounts must be read by a person first, because of a contract clause. The code is still correct, yet the customer’s metric does not budge. That is precisely the kind of failure FDE-Gym’s yardstick is designed to catch.

The way the tasks are built targets the same weakness. Experts took 20 real deployment projects across 18 domains, turned them into customer operating environments, and then changed a few key facts to create variants.

They kept only tasks whose graders had been verified. Because the key facts differ, a familiar solution rarely carries over intact: to pass, an agent has to reread the context of each variant.

Narrow is easy, end to end is hard

Set against recent benchmarks, the boundary is fairly clear. FDE-Bench, by Ding, Zhan, Li and Guo and published on September 23, 2026, measures only a narrow slice of FDE work: configuring deployment environments with Docker, Compose and Kubernetes. On its 136 tasks, seven models solved between 52.9% and 75.0%.

The two benchmarks differ in their tasks and scoring, so they cannot be compared directly. Even so, the trend is plain: the narrower and better specified the task, the better agents perform. When they have to work out for themselves what the customer needs, scores fall sharply.

Another study by Ding and Zhan, “Trains but Doesn’t Learn” (September 21, 2026), treats delivering a post-trained model as FDE work. The agent goes through ten stages, each scored from facts recorded on the platform.

Its central failure mode will sound familiar: a run looks entirely fine, but the final model is no better than the base model.

Which skills are worth more

An open-source project with the same name on GitHub, deeplethe/fde-gym from DeepLethe, builds a full practice platform around the same idea: each exercise is drawn from a real customer handoff and done alongside a coding agent.

The project’s argument is that once AI writes correct code, correct code no longer separates strong engineers from ordinary ones. What still separates them is the ability to produce results when the system is running for real at the customer’s site.

So if you are preparing for an FDE role, do not spend all your time drilling algorithms. Practice customer discovery: in the first meeting, ask how the customer measures success, who has the authority to say “no,” and which data must never leave the system. On your CV, replace lines like “built X” with how the customer’s metric changed.

When reading job descriptions or interviewing, ask directly: is this role accountable for the customer’s outcome, or only for the code it is assigned? The figure of 14 out of 95 says nothing about people, since the benchmark scores only agents. But it shows that the last mile of deployment is where machines are still weakest, and that is where your effort is best spent.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
6 sources