LLM agents playing FDE are good at questioning clients, but only 25% of runs pass acceptance

Sent to interview an enterprise client, an LLM agent does better than many would expect: the strongest model uncovered more than 80% of requirements that had been deliberately hidden. When it came to delivering a working system, the picture was very different. Only 25% of runs passed every acceptance criterion.

Those are the two numbers in “Can LLM Agents Be Forward-Deployed Engineers?”, a study of requirements discovery and delivery for enterprise text-to-SQL systems, submitted to arXiv on 2 November 2026. Marianne Menglin Liu is the first author; her co-authors include Dan Roth and Sujith Ravi.

For anyone working as an FDE, or hoping to become one, the study makes a very practical point: client discovery is not finished just because the list of requirements is long. A requirement recorded with the wrong meaning will still break the delivery.

The yardstick is four months of one person’s work

The authors did not build a toy problem. Their benchmark is a real deployment: an FDE spent four months bringing a text-to-SQL model written for the Oracle dialect into a large enterprise.

The agents had to retrace the two stages that engineer went through: discovering requirements with the client, then delivering a system that passes acceptance.

In the first stage, the team tested 15 models. The strongest cleared 80%, while open-weight models found only 40–50% of the hidden requirements. Looking only at the best result, it would be easy to conclude that customer discovery, the most “human” part of the job, is close to being automated.

Why doesn’t 80% become 80%?

In the delivery stage, the picture changes sharply. Across eight coding-agent configurations, only a quarter of runs met every criterion. According to the authors, the first bottleneck is insufficient discovery: 40% of hidden requirements were missed or misstated.

The two figures do not contradict each other. The 80%-plus mark belongs to the best model, while open-weight models stopped at 40–50%. More importantly, the word “misstated” shows that a requirement can count as found and still not match what the client meant.

Consider a hypothetical example. A client asks for “revenue by quarter”, but at their company revenue must exclude returned goods and the fiscal quarter starts in April. The agent records “revenue by quarter”, yet the SQL it writes is wrong on both counts, and that misunderstanding travels all the way to the acceptance test.

Another benchmark reaches a similar result

This finding does not stand alone. FDE-Bench, published separately by a Baidu AI Cloud team with no shared authors, contains 49 cases drawn from real client projects, in which agents must draw out requirements nobody stated at the start.

Across 18 agent configurations, the best mean delivery score was only 0.5510 on a scale of 1.0. The Baidu team concluded that the agents tested cannot yet reliably deliver end to end at FDE level.

Two independent studies, using two different datasets, both found agents falling short at delivery. That makes it hard to dismiss the 25% figure as a quirk of one experiment.

Humans still have to keep two jobs

The advice from Liu and her co-authors is specific: humans should review the list of requirements the agent has found, and monitor the agent’s plan while it executes. On this view, the FDE moves into a supervisory role rather than being replaced.

The skill worth practising now is following a requirement from the client meeting all the way to the acceptance test, and writing down the business definition behind every term the client uses. That is exactly where the authors say human review is needed.

When reading job descriptions, look for phrases such as “requirements gathering” or “acceptance criteria” alongside “LLM agents”. On a CV, a project that states how many requirements were discovered, how many acceptance criteria there were and how many were met is more persuasive than any adjective.

Agents have learned to ask questions. Checking whether the answers were understood correctly is still the engineer’s job.