Your model can change while your code stands still: how to catch it before the customer does

Picture a Tuesday morning. A logistics customer sends a message: “The invoice reading system has been getting far more wrong this week.” You open the git log. No commits for three weeks, no prompt changes, no schema changes.

By then you are already a beat behind: the customer knew before you did, and you have no number to answer with.

The skill to build comes down to three connected tasks: pin versions, measure drift, re-run evals. Done properly, you are the one who messages the customer first, with a number attached.

The code stands still, but the system can still change

The evidence has been around for some time. Comparing the March and June 2023 versions of GPT-3.5 and GPT-4, the authors Chen, Zaharia and Zou concluded that the behaviour of the “same” LLM service can change substantially in a short time, and that LLMs need continuous monitoring.

OpenAI’s documentation is more blunt: LLM output is inherently non-deterministic, and model behaviour changes between snapshots and between model families. But not every drop in quality comes from a model change. In a postmortem published in September 2025, Anthropic confirmed that three infrastructure bugs had intermittently degraded the quality of Claude’s responses.

So “the provider changed the model” is usually a hypothesis, not a diagnosis. To test it early, you need a canary.

The word originally described rolling a change out to a small group of users before extending it across the whole infrastructure. Here it is borrowed for a small test set that runs regularly through the day and raises the alarm before the customer has time to see the fault.

Separating three cases shows when a canary is enough and when something else is needed:

What happens	Does pinning a version help?	How to detect it
A generic name (alias) starts pointing to a new snapshot	Yes, because model changes are then in your hands	A canary if not yet pinned; evals on every change when upgrading a pinned version
A model is about to be retired	Only until the retirement date	Notices, a usage inventory, early testing of the replacement
A provider-side infrastructure bug	No, because the infrastructure around the model can still change	A canary running regularly on production

Pinning a version is a first step, not a shield

Start by distinguishing two kinds of model name. In Anthropic’s documentation, a snapshot is the fixed model version itself, while an alias such as claude-sonnet-4-5 is a convenience name pointing to the most recently dated snapshot of that minor version.

The cheapest move, then, is to call the model by its fixed snapshot name if the provider offers one, and to keep that name in exactly one configuration file. A simple rule: if a change in behaviour would force you to investigate, pin. From then on, every model change is a commit with someone accountable for it, rather than something that happens out of sight.

Anthropic itself spells out this limit: each ID keeps the same underlying model throughout its lifetime, but the serving infrastructure around the model can still change over time. The company also treats infrastructure as the most likely cause when a stable ID suddenly produces unusual results.

A pinned version also has an expiry date. Anthropic commits to giving at least 60 days’ notice before retiring a publicly released model, so test your application against the replacement early, on your own tasks.

Sixty days sounds long, but on a real project it is not much. If the customer allows a release only every 30 days, you get about two attempts, and the first should be spent finding problems rather than finishing the fix. With other providers, read their policy rather than assuming this figure applies.

The first job on receiving a notice is to inventory usage by API key and model. You are looking for the overnight job or the API key that nobody on the team still remembers.

A worked example with 200 invoices

Back to the logistics customer. Take 200 real invoices, because an eval only means something if it resembles what the customer sends in every day. Split them into two groups: 50 hard-to-read scans and 150 clean PDFs. Score each case field by field, such as ID number, date and total amount, because exact string matching will raise constant false alarms on non-deterministic output.

The hardest part is the correct answers, and you should not settle them alone. Ask the person who reconciles invoices at the customer to confirm each field; if the customer has an accounting system, pre-filling from it for them to review is faster. Where two people read a case differently, let the business owner decide, then write down the rule.

Before measuring drift, you must know how much the score normally varies, for both the total and each group. Run the whole set three times on the same pinned version. Suppose you get 186, 188 and 190 correct cases; within that, scans score 45, 46 and 47, and clean PDFs 141, 142 and 143.

Take the lowest score and subtract the range of variation to get the threshold. The total varies by 4 cases, so its threshold is 182; scans and clean PDFs each vary by 2 cases, so their thresholds are 43 and 139 respectively. An alert fires only when a score falls below its threshold.

Suppose that this morning the canary raises the alarm and the 200-case set re-runs automatically. The result is 176 correct, down from 94% to 88% against the baseline of 188. The code has not changed and the number is outside normal variation, so this is a real signal.

Only by looking at each group do you see where the damage is: scans fell from 46/50 to 36/50, below the threshold of 43, while clean PDFs went from 142/150 to 140/150, still above the threshold of 139.

Now try a harder scenario: scans fall to 40/50 while clean PDFs hold at 142. The total is 182, exactly at the threshold rather than below it, and if you looked only at the total you would see nothing. That is why each group needs its own threshold:

# Model name, case counts and thresholds are for illustration only
PINNED_MODEL = "vendor-model-2025-06-01"

BASE_RUNS = {
    "total": [186, 188, 190],
    "scan": [45, 46, 47],
    "pdf": [141, 142, 143],
}

def threshold(runs):
    return min(runs) - (max(runs) - min(runs))

def check(base_runs, today):
    return [name for name, runs in base_runs.items()
            if today[name] < threshold(runs)]

check(BASE_RUNS, {"total": 182, "scan": 40, "pdf": 142})  # ["scan"]

In this scenario, check returns only the scan group, because 40 is below 43, while the total has not triggered an alert. Three runs give only a rough estimate of variation, especially for a group of 50 cases, so run more if you can, and re-run any group that alerts to confirm before telling anyone.

Nor is this formula a general rule. Ask the customer which group hurts most when it goes wrong, then set the threshold tighter there.

The eval set must run on production

An eval set that lives only in a notebook cannot raise an alarm on its own. The most valuable lesson in Anthropic’s postmortem is that the company’s own evals did not catch the degradation users were reporting, and one of its remedies was to run evals continuously on the real production systems.

A provider’s evals do not measure your customer’s task, so they cannot replace yours. And because faults appear intermittently, a once-a-week run can easily miss them.

So build two tiers. The fast tier is a canary of about 30 cases covering all the groups, run several times a day through the same API key, gateway and prompt as production, with its own thresholds measured from this set, because small samples vary more.

The full tier is the 200-case set, run automatically every night and as soon as the canary alerts, to confirm the signal and show which group is affected. It also runs on every change, as OpenAI recommends, and serves as the automated regression suite before any prompt or model change.

When an alert fires, open each failing case before saying anything to anyone. If the score has dropped on the same pinned version with no code change, send the provider the reproducible cases and, if the customer agrees, move the affected group to a human reviewer in the meantime.

The message to the customer should contain numbers and an action: “Since 9am today, automated checks show scanned invoices correct in 36 of 50 cases, compared with 46 of 50 before; clean PDFs are not affected. The deployment team proposes moving scans to manual review during the investigation, and will update you at 2pm.”

Three mistakes that cost you your lead over the customer

The first mistake to avoid is rushing to conclude that the model itself has changed. In the 2025 incident, Anthropic attributed the cause to infrastructure bugs rather than a deliberate model change, so anyone who guessed “the model was changed” at the time guessed wrong, even though the fault did lie with the provider.

Whatever the cause in your case, the customer needs reproducible cases, not a hypothesis.

The second mistake is building the eval set from tidy examples your own team made up, then leaving it untouched. Customer data changes over time, so each time the customer reports an error, add that case to the set.

The third mistake is leaving the canary without an owner after handover. An alert that nobody receives is no alert at all, and the customer is once again the first to notice.

What to ask in interviews, what to put on your CV

When reading an FDE job description, look for terms such as regression eval, model upgrade and production monitoring. Whether or not the description mentions them, ask at interview how their most recent model change went, and who spotted the first fault.

On your CV, write one line with a specific number rather than just listing tools. For example, if it is true of you: “Built a 200-case eval set from customer data, with a production canary, detecting regressions before the customer reported them”.

This week’s exercise: take 30 real inputs from one of your projects, write the answers yourself, run them three times on the same snapshot and record the variation. Then run them on a different snapshot and see whether the difference exceeds that variation. The next time someone asks “has the model changed?”, you will answer with a number instead of a feeling.