FDE Times dataFDE jobs open 924▲33 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 80Companies hiring 196Remote-friendly 25%Median US pay $200kTop hirer Databricks 107
中文VI

The newspaper of the Forward Deployed Engineer

Guides

The ten seconds after an AI demo that shape the whole engagement

A prototype that reads three hand-picked invoices is not a system for every invoice, and the finance chief watching cannot see the gap.

ShareLinkedInFacebookX

In brief

  • A demo shows the system can do the job. It does not show the system will do it every time. Production has to be right every time, including on data nobody has seen yet.
  • Demo on the client's real data, show a failure case before anyone finds it, and quote an accuracy figure only when an eval set stands behind it.
  • Present the prototype as evidence that the next phase is worth doing, not as a product that is nearly finished.
Infographic of five cards in sequence, showing a 20-minute AI prototype demo script. 01 Set the scope: this is a prototype, run on 50 invoices the client sent yesterday. 02 Real data: run two or three of the most common invoice types live. 03 Show failures: a blurry photo makes the system misread the total. 04, highlighted in orange, Eval figure: 41 of 50 invoices extracted correctly in every field; the 9 errors are mostly tilted photos. This is where most people go wrong. 05 Next steps: processing time, cost per invoice and a trial plan in a sandbox. The strip below answers "can it run next month?": next month it will not yet run company-wide; run in parallel with the accounting team in a sandbox, test on new data, and only then decide whether to scale.
An honest demo sets expectations from the start, shows the client failures unprompted, and uses the eval figure behind it to answer "when will it be done?".

Picture this. On Monday the head of accounting at a logistics company complains that her team loses whole afternoons retyping figures from supplier invoices. On Tuesday you open your laptop and show her a prototype that reads the invoices and fills in the spreadsheet itself.

Heads nod around the room. Then the finance director asks the question every FDE meets sooner or later: “So it can run across the whole company next month?” Your answer in the next ten seconds will set the mood for the months of work that follow.

Listen today, demo tomorrow: that is the rhythm of the job. Shilpa Balaji, a former FDE at Palantir, describes building a prototype for whatever the client said one day and bringing it back to show them the next.

Working that fast makes it just as easy to promise fast. A prototype looks finished long before it is, and fde.academy puts overpromising among the most common mistakes new FDEs make.

What does a demo actually prove?

Your prototype has just read three invoices correctly. You chose those invoices, and you knew in advance who would be in the room. That is all a prototype needs to do: work once, on hand-picked data, in front of a prepared audience.

A company-wide system is a different thing. It has to read every invoice correctly, including ones nobody has seen yet. The finance director cannot see that gap. She sees a screen that runs smoothly.

Demos are built to show capability, not to prove reliability. A post on the EB Pearls blog argues that projects rarely stumble between demo and production for technical reasons. More often, the two sides expected different things. So your main job in the meeting room is not to impress. It is to help the client set realistic expectations.

To do that, treat an impressive prototype as evidence that the expensive phase behind it is worth starting. A realistic plan sees it that way, not as a product that is almost done. Your answer to the finance director should start from there.

Why demo on the client’s own data?

First Round Review reports that at Looker, demos always ran on the client’s real data, so every demo doubled as a proof of concept. Looker had no sales demo built on fake data. This is not only a matter of ethics: real data forces the prototype to meet, straight away, the things that will break it later.

The second reason matters even more. What FDEs find on site is often very different from what the contract sold. A tidy set of sample invoices only tests the assumptions written into the contract.

Real invoices come as photos taken at an angle, with handwritten notes, from suppliers who print the total in an odd place. Those are the documents that tell you what the client actually needs.

So your first task on site is to ask for a batch of real samples, including the “ugly” ones the operations team dreads most. If the client hands over only clean samples, ask directly: which type of document costs the team the most time?

A 20-minute script, part by part

Back to the meeting room at the hypothetical logistics company. The table below is a script you can use almost word for word; just swap in the business process.

Part What you say or do Purpose
Opening: set the scope “This is a prototype, running on the 50 invoices you sent yesterday. I’ll show you what it can do and where it falls short.” Set expectations before the client sees anything
Run on real data Open two or three invoices of the most common type and run them live The client sees their own process
Show a failure case yourself Run a blurry photo that makes the system misread the total The client sees you are not hiding weaknesses
A number backed by an eval “41 out of 50 invoices had every field extracted correctly. The 9 errors were mostly photos taken at an angle.” A number the client can check for themselves
Trade-offs and next step Cover processing time, cost per invoice and the plan for sandbox testing Turn “when will it be done?” into a plan

The fourth part is where many people go wrong. The reason is simple: without an eval set, nothing will tell you when output quality starts to slip.

Hamel Husain goes further: in his view, failed AI products almost always share one root cause, a failure to build a robust enough evaluation system. The figure of 41 out of 50 has value because the client can open each invoice and check it.

The fifth part needs the same candour. Agent systems usually accept higher latency and higher cost in exchange for better results, and the finance director needs to know that before signing off. The sensible next step is thorough testing in a sandbox, with appropriate guardrails, before the agent runs for real.

By now you have your answer to the “next month” question:

“Not company-wide next month. The prototype shows automation is feasible for standard invoices. The next step is to run it alongside the accounting team in a sandbox, measure it on new data, and only then decide whether to scale up.”

How to prepare before the demo

Start with the data, not the slides. Get real samples from the client, then label the correct answers by hand for a few dozen of them. It is tedious, but it turns a sense that things “look fine” into a number you can defend.

Next, build the simplest version that works. Anthropic advises adding complexity only when it is truly needed. That also keeps the demo honest, because what you show is exactly what you built, with no layer added just to look impressive.

Finally, run the eval and group the errors by cause. Pick one representative error to include in the demo, and write out your opening line and your answer to “when will it be done?” in advance. If reading that answer aloud makes you wince, it is probably still overpromising.

Mistakes that turn a demo into a promise

A common mistake is demoing on sample data picked to look good. The client will remember that smooth-running system. The first time it stumbles on a real invoice, they will feel misled.

The second is hiding the errors. In fact, a failure you point out yourself builds a credibility that a flawless demo cannot.

The third is saying “about 95% accurate” with no eval set behind it. The fourth is building a multi-step agent just to impress, then having to carry its latency and cost. The fifth is answering “when?” with a specific date in the meeting room.

All five share one root: treating the demo as the destination. fde.academy writes that an FDE’s success is measured by whether the client’s real problem gets solved, not by how impressive the demo was.

Showing this skill on your CV

When reading a job description, check whether “prototype”, “customer-facing demo” and “evaluation” appear together. If they do, the role needs exactly this skill.

On your CV, one tight line is enough, for example: “Built a document-extraction prototype on the client’s real data; built a labelled eval set before the demo, presented failure cases alongside results and proposed a sandbox trial phase.”

That line tells the reader you work fast and still know where to stop. fde.academy notes that clients remember the people who kept their word far longer than the people who sounded confident in the meeting room. The first demo is your first chance to keep yours.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
6 sources
Read next on the roadmap · Stage 4: CustomersA new FDE's six weeks to show value is really about fourInternal onboarding eats the first fortnight, so the plan must lock in one small, high-impact task with the customer before the week-five wall.