If you never measure the 'before', you can never prove the 'after'
Five days spent timing the client's manual process give a forward deployed engineer the numbers to defend a project when the ROI question comes.
In brief
- The most common reason an AI ROI case goes nowhere is that nobody measured the 'before' state
- A baseline must capture variance, track a single metric and be locked before the pilot starts
- The FDE, sitting next to the client, is best placed to time the manual process in the first week
HatchWorks, in its guide to measuring the ROI of forward deployed engineers, names the most common reason an AI ROI case goes nowhere: nobody measured the “before” state. The model was fine and the code ran. Nobody could prove it.
The question usually arrives around the sixth week of a pilot. The client’s sponsor asks something perfectly ordinary: how much time does this tool save the team? You know it is faster, and so does everyone on the team. Faster than what, though, you cannot say in numbers.
For an FDE, the gap can be closed with one job done in the first week, before a single line of code: timing the manual process the client runs today. What follows explains how to do it, with a worked example, and the mistakes that make a baseline useless.
A baseline is a snapshot, not just a number
The first rule HatchWorks sets out is to lock the baseline before the engagement starts. Unframe, writing about the ROI of AI pilots, likewise advises establishing a baseline before the pilot is deployed, and defining the ROI metrics and the reporting cadence before day one.
The reason is simple: once the new tool is running, the old process disappears and you cannot go back to measure it.
As for what to measure, HatchWorks recommends the workflow metric the first build is aimed at, and one of them, not ten. Then record that metric as it actually behaves, variance included, so you know what “normal” looks like.
Variance is the part most often skipped. If the old process has fast weeks and slow weeks, one good week after launch proves nothing. Without variance data, you cannot tell a real improvement from a lucky week.
Why the FDE should do the measuring
The essay “What is a Forward Deployed Engineer, Anyway?” describes the FDE as someone willing, even insistent, to spend a lot of time with customers, working closely with them to get deep into the problem. That position makes you the right person to measure. Managers on the client side tend to remember only an estimate, and the people doing the work are too busy to keep records themselves.
There is a further benefit. Rocketlane describes OSV, a team that ran a project health-scoring framework by hand for five weeks before writing the first line of code, on the principle of proving the framework manually before automating it.
According to Rocketlane, that manual phase is what made the automation trustworthy. Your baseline week works the same way: you measure the process and learn it from the inside at once.
Example: a week of complaint emails
Take a hypothetical delivery company with four staff handling complaint emails. The first build is an agent that reads each email, classifies it and pre-fills a ticket.
The single metric you agree with the sponsor: minutes from the moment a staff member opens the email to the moment the complaint ticket is saved in the system. The definition has to be that precise, because a vague “handling time” will mean something different to everyone.
For the first two days, you sit next to each person and time them yourself. For the next three, the staff log their own work in a simple spreadsheet while you spot-check a few rows. An extract of the log looks like this:
| ID | Type | Start | Done | Minutes | Blocked by |
|---|---|---|---|---|---|
| C-07 | Broken item, photo attached | 09:12 | 09:19 | 7 | — |
| C-08 | Lost parcel | 09:20 | 10:04 | 44 | Waiting for warehouse confirmation |
| C-09 | Wrong address | 09:31 | 09:42 | 11 | — |
| C-10 | Broken item, no photo | 10:05 | 10:31 | 26 | Emailing customer for details |
| C-11 | Lost parcel | 10:40 | 11:38 | 58 | Waiting for warehouse confirmation |
Say that by the end of the week you have 40 rows, with a mean of 19 minutes, a median of 14 minutes and a p90 of 38 minutes. The mean is dragged up by a long tail made up almost entirely of “lost parcel” cases waiting on the warehouse. Report only the 19 and you have buried the most important finding of the week.
Now read the data the way you will when reporting results. If the agent only speeds up the easy cases, the median could fall sharply while the p90 stays stuck at 38, because the bottleneck sits in the warehouse rather than in reading emails.
Knowing this in week one, you promise the client only what you can actually improve, and you already have the evidence if you want to extend the scope to communication with the warehouse.
Splitting by day teaches a further lesson. Suppose Monday’s median is well above the other days because emails pile up over the weekend. If you later measure only midweek, you will inflate the results without meaning to.
A baseline in five days
On day one, sit with the sponsor to agree a single metric, with its start point, end point and where the data will be recorded. On day two, time the work yourself while sitting next to the people doing it, noting the reason for every block. From day three, hand the spreadsheet to the client’s team and cross-check random rows to keep the data clean.
At the end of the week, calculate the median and p90, then break them down by task type and by day. Write one page for the sponsor setting out the number, the variance and the reporting cadence both sides have agreed, for example once every two weeks.
Where conditions allow, Unframe suggests a control group: two staff using the agent and two still working by hand over the same period. With a control group, nobody can claim the improvement came from a quiet week.
Mistakes that make a baseline useless
The most common mistake is asking instead of measuring. “Each email takes about fifteen minutes” is the respondent’s memory, not data, and it usually leaves out the long tail entirely. The second is tracking a dozen metrics at once, so that none of them ends up with enough data to support a conclusion.
The third is harder to spot: changing the process while you measure it. Suggest a small shortcut to the staff on day two and the rest of your data is no longer “before”. Write every improvement idea in a notebook and save it for the following week.
The last is measuring after the demo has already been tried. Once half the team has touched the new tool, the baseline is contaminated. That is why HatchWorks and Unframe both stress the words “before the start”.
One line on a CV beats ten features
A baseline also gives you something concrete for your CV. Instead of “built an email classification agent”, write something like: established a 40-case baseline in week one, cut median handling time from X to Y minutes.
In FDE interviews, the story of how you chose the metric and found the long tail shows the listener you can prove value with numbers as well as with a demo.
When reading job descriptions, treat phrases such as “measure impact”, “outcomes” or “prove value” as the places to bring your baseline story into your application and your interviews.
The sixth-week question will come. If you measured from the first week, the answer is one spreadsheet away.
Was this article useful?
Thanks for the feedback!