Eric Ries's The Lean Startup: the book that turns an FDE's AI pilot into an experiment with a stopping rule

Picture a familiar scenario. The demo goes well, the customer nods, and then week follows week without anyone willing to say whether the pilot has succeeded or failed. In that scenario the problem is not the model. It is that nobody wrote down, at the start, what failure would look like.

Eric Ries wrote The Lean Startup for company founders, not for engineers deploying AI. The US edition, published by Crown Business, carries the subtitle “How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses”; the UK edition from Portfolio Penguin, released on 6 October 2011, uses “How Constant Innovation Creates Radically Successful Businesses”.

Penguin says the book has sold more than a million copies. For an FDE, the reason to reread it lies elsewhere: it trains exactly the skill needed on a customer site, which is designing a pilot as an experiment with a hypothesis, a measurement and a stopping point.

What loop sits behind the book?

The core of the method is the Build-Measure-Learn loop. According to the principles Ries has published, the fundamental activity of a startup is to turn ideas into products, measure how customers respond, and then learn from that whether to pivot or persevere. Replace “product” with “AI pilot” and you have an FDE’s working process.

The Principles page also states plainly that you do not have to work in a garage to be in a startup. In other words, an FDE deploying an agent in an insurer’s offices is doing exactly the kind of work the book describes.

First idea: progress is what has been validated, not what has been shipped

Ries proposes validated learning as the measure of progress. For an FDE, this changes how a weekly report is written. “Finished the API integration and built the RAG pipeline” shows you were busy; it does not show what you learned.

Imagine a pilot of an agent that triages support tickets for an insurance company. A validated-learning report would read: “After two weeks, of 500 tickets in the group under test, the agent correctly handled 120 without human intervention.” That number tells you whether an assumption is true or false; a list of completed tasks does not.

Second idea: build only what is needed to measure

If the loop starts with Build and the purpose of Build is to Measure, the first build only needs to be enough to measure one assumption.

In the example above, the assumption worth testing is not “an LLM can read a ticket”, which is almost certainly true. The real assumption is that the customer’s support team will trust the agent’s output enough to drop the step of checking it again.

So the first build might be nothing more than a panel of suggested classifications placed beside the ticket queue, with metrics on how many suggestions staff accept. Building more than that is wasted effort before you know whether the work is worth doing.

Third idea: the decision to stop must be written down in advance

Every Build-Measure-Learn loop ends in a decision: pivot, meaning change direction, or persevere, meaning carry on. The book’s official page describes it as offering a method for making this decision.

That decision only has value when paired with innovation accounting, which the Lean Startup principles call “the boring stuff”: how to track progress and hold innovators accountable for results. Tristan Kromer of Kromatic makes this concrete with two metrics set in advance: a success criterion and a failure condition.

The reason, according to Kromer, is that setting clear criteria before the evidence appears reduces bias once the data has been seen. He also argues that the decision has four outcomes, not two: besides pivot and persevere there are scale and kill.

Back to the example. Before starting, you and the customer agree that after two weeks the agent must handle at least 40% of tickets in the test group on its own for the pilot to expand, and that below 20% it stops. The result of 120 out of 500 tickets over those same two weeks is 24%: not enough to scale, but not low enough to trigger the kill condition.

At that point there is no more arguing on gut feel. You pivot, for instance by narrowing to one ticket type the agent handles well, then run another two-week round with the same measurement.

Where do stopping rules go wrong?

The most common mistake is moving the threshold after seeing the number. The result comes in at 24%, someone suggests that “actually, 20% is already good”, and the 40% threshold quietly disappears. This is precisely the bias Kromer wants to block: once the data is visible, everyone has a reason to read it the way they want.

The second mistake is sliding back into counting deliverables. When the numbers are not flattering, a report easily drifts into “added three connectors, tuned the prompt”. Those lines may be true, but they do not replace an answer to the assumption, and that answer is exactly what validated learning demands.

The third mistake is harder to spot: the threshold and the result are measured over different time windows. Set a threshold for two weeks but report one week’s figures, or the other way round, and comparing 24% with 40% becomes meaningless.

A sensible safeguard is to email the customer the agreed thresholds, the time window and the four outcomes before the pilot runs, so nobody can change them afterwards.

How to read it, and who should

The book suits engineers who have run at least one pilot that dragged on without end. Read the sections on the Build-Measure-Learn loop and validated learning before meeting the customer, the innovation accounting section when drafting the pilot plan, and the pivot-or-persevere section before the mid-point review.

The official material speaks the language of startups, so you have to translate “the startup’s customer” into “the end user at the client company” yourself. If your customer is a large corporation, go on to The Startup Way (2017), Ries’s follow-up, which Penguin Random House describes as aimed at enterprises such as GE and Toyota.

On the career side, this skill is easier to demonstrate than you might think. When reading an FDE job description, check whether the role makes you responsible for scoping pilots and setting success criteria; if it does, put this skill at the top of your CV.

A line such as “set a 40% threshold, measured 24%, narrowed scope and ran a second round” carries more weight than ten lines listing technologies.

Exercise: write an experiment sheet for your pilot

Take the pilot you are working on or have just finished, and write five lines on a single page. Line one is the riskiest assumption, expressed as end-user behaviour. Line two is the metric you will measure and the single time window that applies to every number.

Line three is the threshold to scale, line four the condition to kill. Line five records in advance which way you will pivot if the result lands in between. Send the page to the customer before any data arrives, and when the results come in, compare them only against that page.

A good FDE is not someone who never stops a pilot, but someone who knows from day one when they will stop it.