Check the customer's data before you commit to an AI project

In your second week at the customer, after a smooth demo, the head of operations asks: “So, can we be in production in three months?” The agent in the demo answered almost every question correctly. But it ran on a CSV file that an analyst had cleaned up for you the Friday before.

Whether you answer “yes” or “not yet” will shape the rest of the engagement. A post on the Varsity blog argues that an FDE’s work starts with real production data, with security reviews and access permissions. And production data is messy, locked behind permissions, and held by people who have no particular reason to help you.

Discover that halfway through the build and every earlier promise becomes a liability. So the FDE’s job is to answer the data question before signing a commitment. It can be done in clear, deliberate steps.

Why does the customer’s dashboard mislead you?

The first obstacle is the customer’s own confidence. Easy.bi observes that most mid-sized companies believe their data is good simply because their dashboards keep working, and then says it bluntly: that does not mean the data is ready for AI. Veeam draws the same line: usage describes activity, while readiness describes capability.

The reason is easy to picture. A dashboard rolls thousands of rows up into a single number, so a few hundred rows with missing data disappear. An agent that has to answer individual questions one at a time will trip over exactly those rows.

The dashboard hides something else too: readiness is always tied to a specific use case. The same dataset can be enough for one problem and fall short for another. So do not score “the customer’s data” in general; score “the customer’s data for task X”.

Phos AI Labs sets out five properties of AI-ready data: accessible, consistent, complete, accurate and current. Easy.bi uses a different set of 4 criteria, each scored on a 1–5 maturity scale, during a 5-day assessment.

Phos’s five properties are enough to examine the data itself, but they do not tell you whether you are allowed to use it. That question gets its own section, right after the example below.

A worked example, start to finish

Imagine a spare-parts distributor that wants an agent to answer two questions from its dealers: “is this part in stock?” and “when will it be delivered?” Its inventory dashboard has run fine for years. You write the use case as one sentence: the agent must give the correct stock level and expected delivery date for each order, at the moment the dealer asks.

Next, you ask for direct read access to the ordering system rather than an export file, and pull 200 orders at random. Hypothetical results: only 124 orders have an expected delivery date. Two warehouses use different product-code conventions, and the data is exported by hand, by one person, each night.

Phos scores each checklist item as Yes/Partial/No (2/1/0 points), sums the points per criterion, and treats any criterion below 70% as a gap to close before building. RaftLabs, another vendor, also uses 70 as a cut-off, but out of a total of 100, so treat the figure as vendor experience rather than an industry standard.

Take the “complete” criterion as an example, and suppose its checklist has three items. Every order has a part code: Yes, 2 points. Stock is recorded per warehouse: only one warehouse does this fully, so Partial, 1 point. Orders have an expected delivery date: 124 of 200, or 62%, so Partial, 1 point.

That totals 4/6 points, about 67%, below the 70% threshold. Note that the 62% figure is not a score: it is only the evidence for choosing “Partial” over “Yes”. The two are different measures, and mixing them up is the fastest way for a scorecard to lose credibility with the customer.

The hypothetical scorecard for all five properties looks like this:

Property	How to check	Hypothetical result	Score
Accessible	Direct read access, or does someone have to export it?	Manual export each night	No (0), gap
Consistent	Does the same part code match across both warehouses?	Two different conventions	No (0), gap
Complete	Three items: part code, stock per warehouse, delivery date	Delivery date on 124/200 orders	4/6, below 70%, gap
Accurate	Reconcile a sample against a physical stock count	Needs customer confirmation	Not yet scored
Current	Lag relative to when the dealer asks	Up to one night behind	No (0) for this use case

The “current” row shows why scoring must be done per use case. If the customer wanted a monthly demand forecast, data that is a night old would be fine. For an agent answering in real time, that lag is enough for it to report the wrong stock level.

Is one failing criterion enough to stop?

Yes, and that is the crux. Phos’s threshold applies to each criterion, not to the average, so four good rows cannot make up for one bad one. Veeam puts the same idea another way: the ceiling of AI readiness is always capped by the floor of data readiness.

The Revenue Experts newsletter applies this logic to budget approval: every AI initiative must pass four core data checks, and failing any one marks it as not ready, however well it scores elsewhere. One of the four is that data quality must have been checked within the past 90 days.

For the spare-parts distributor, the simplest question you can send is: when did anyone last check the delivery-date field? If nobody can answer, the nightly export and the 62% figure have answered for them.

Ask about governance in week one

The Varsity post points out that production data is permissioned and owned. In the example above, ask: may each dealer’s contract pricing appear in the agent’s answers? Who signs off on that?

If the answer is the legal department and they have never heard of the project, you have just found your biggest risk. The same post stresses that discovery happens with the people who run operations, so do not only ask the person who signed the contract.

Bring three questions to the first discovery session: who owns the data, who approves the model reading it, and which fields count as personal or sensitive data. Asking in week one costs an email. Finding out in week ten can cost the whole project.

How to do it yourself, and the common mistakes

The process fits into one week. Write the use case as one sentence, ask for read access to production rather than an export file, sample the data and score the five properties, then ask the governance questions. Finally, write a one-page go/no-go memo listing each gap, the work needed to close it and who owns it.

The most common mistake is trusting the customer’s dashboard. RaftLabs reckons most organisations rate their data maturity 2-3 levels higher than it really is, and advises tackling the weakest criterion first. The second mistake is scoring data in general, without tying it to a use case.

The opposite mistake is real too: waiting for perfect data. Easy.bi finds most mid-sized companies sit at level 2-3, and warns that anyone who waits for level 5 before starting will never ship.

The worst mistake is running the assessment after signing a commitment. By then every gap you find looks like an excuse rather than a valuable finding.

Exercise for this week

Take a dataset you already have access to and assign it two imaginary AI use cases with different rhythms: one that needs instant answers, one that only needs a monthly report. Score the five properties for each use case with 2/1/0 points, then circle the rows where the conclusion changes.

Write the result as one line with numbers, along the lines of “assessed data for an agent use case, found the delivery-date field present on only 62% of orders, recommended closing the gap before building”. That line belongs on a CV, and it is the story to tell when an FDE job description mentions scoping, discovery or data assessment.

Next time someone asks “can we be in production in three months?”, you will ask “for what?” before answering.