# A PSI of 0.23 does not tell you to retrain the model

> Retraining should follow performance on fresh labels against thresholds set from the customer's own history, or a sprint gets burnt on a healthy model.

Original: https://fdetimes.net/en/guides/psi-drift-retrain-decision-thresholds/

Three weeks after go-live, the customer's monitoring dashboard turns amber. An important feature is drifting, and the customer's product manager sends you a single line: "Do we need to retrain the model?"

Every FDE faces this question after a deployment. Answer "yes" too quickly and you burn a sprint retraining a model that is still working well. Answer "not yet" without evidence and, sooner or later, the customer will discover for themselves that the model has been wrong for a month.

This guide shows how to answer that question with numbers: how to read drift signals, which thresholds call only for logging, which call for investigation, and when retraining is genuinely worth doing.

## Drift is an alarm, not a diagnosis

The first thing to grasp is that there are several kinds of "drift", and each needs its own response. Google separates two concepts. Training-serving skew is when production data differs from the data used for training. Inference drift is when the distribution of features in production changes over time.

Evidently adds a third definition, concept drift: the relationship between inputs and the target changes, as distinct from the input data itself changing. Because what shifts is the relationship rather than the inputs, incoming data can look exactly as before while "correct" has come to mean something else. That is why this kind is so easy to miss.

| Type | What changes | How to detect it | First thing to do |
|---|---|---|---|
| Training-serving skew | Production data differs from training data from the outset | Compare the production distribution with the training set | Review the feature pipeline, since the fault usually lies there rather than in the model |
| Inference drift | The input distribution shifts gradually over time | Compare time windows against each other (PSI, distribution distance) | Investigate the cause and check performance on new labels |
| Concept drift | The relationship between inputs and target | Only becomes clear once real labels are available | Retrain on newly labelled data |

The table makes a point many people overlook: tools that measure drift on inputs directly catch only the first two types. Concept drift can be seen only with labels.

## Calculate a PSI yourself

PSI (Population Stability Index) is the most widely used measure of how far a feature has shifted. Arize gives three "rule of thumb" bands. Below 0.1 is a small shift, which usually needs only routine logging.

From 0.1 to 0.25 is a moderate shift: investigate the features and check the model's performance on recent labels. Above 0.25 is significant drift, and the model should be revalidated or retrained.

Imagine your customer is an e-commerce marketplace using a fraud-scoring model. You split the "order value" feature into 4 bins. In the training set, each bin holds 25%. After three weeks, production splits 10%, 20%, 30% and 40% respectively: high-value orders have increased.

```python
import numpy as np

def psi(ref_pct, cur_pct, eps=1e-6):
    ref = np.clip(np.asarray(ref_pct), eps, None)
    cur = np.clip(np.asarray(cur_pct), eps, None)
    return float(np.sum((cur - ref) * np.log(cur / ref)))

print(psi([0.25, 0.25, 0.25, 0.25], [0.10, 0.20, 0.30, 0.40]))
# ≈ 0.228
```

Bin by bin, the contributions come to roughly 0.137, 0.011, 0.009 and 0.071, summing to about 0.228. On Arize's bands, that falls in the "moderate" zone. The right answer for the product manager at this point is "not retraining yet, but we need to check now", not "yes".

Most of the value comes from the lowest bin, where the share fell from 25% to 10%. That is the clue for starting a conversation. Perhaps the customer has just run a promotion on expensive goods. Perhaps a low-price sales channel was switched off, or the pipeline is filtering out small orders.

## A tool's default threshold is not the customer's threshold

Here lies the second trap. Every monitoring tool ships with its own default number. For tabular data with more than 1,000 reference rows, Evidently defaults to Wasserstein distance for numerical columns and Jensen-Shannon divergence for categorical columns, with a threshold of 0.1. At whole-dataset level, `DatasetDriftPreset` flags drift only when half the columns have drifted.

Vertex AI Model Monitoring v1, by contrast, monitors all categorical and numerical features by default with a threshold of 0.3. The three numbers 0.1, 0.25 and 0.3 are measured with different rulers, so setting them side by side for comparison is meaningless.

Carry a threshold from one tool to another and your alerts will either fire constantly or never fire at all.

Arize itself says the PSI bands are only rules of thumb, and recommends calibrating the false-alarm rate on historical data. On the customer side, the method is fairly simple.

Compute PSI for each past month in which the model was running well. If every month containing Tet or Black Friday also comes out at 0.2, then 0.228 in that season is normal.

**Key point:** The best drift threshold is one that has been tested against the customer's own past.

## Performance, not drift, decides retraining

Google's MLOps guidance lists several kinds of retraining trigger: on a schedule, when new data arrives, and when model performance drops markedly. The last is closest to the customer's question. Drift on inputs only tells you where to look. Performance tells you whether the model is actually failing.

Evidently, however, points out a big limitation: retraining requires newly labelled ground-truth data. For a fraud model, labels can arrive weeks later, once refund disputes have been settled. In that gap, use proxy metrics as an early warning, such as the share of orders blocked or the distribution of scores the model returns.

As for how often to retrain, Google advises basing it on how fast the data changes and what each retraining costs. A demand-forecasting model for a retail chain and a fraud model being probed by bad actors cannot share a schedule.

## For LLMs, evals replace PSI

If what you deployed is a fine-tuned LLM or an agent, most of the same thinking holds. Only the measure changes. OpenAI's optimisation loop starts with evals and repeats continuously, running on test data that represents real inputs.

So the decision to fine-tune again should rest on eval scores, run on a periodic sample of production prompts. Users' sense that "the answers have got worse lately" is not evidence. When customers start asking about a new product line, add those questions to the eval set first, and only then consider retraining.

## A procedure for the next alert

When the dashboard turns amber, work in this order. First, establish whether this is skew or drift. If production diverged from the training set from day one, fix the pipeline.

Next, read the PSI or distance metric against the threshold calibrated on history, not the default. Then find which value range contributes most, and take that question to the business team.

After that, check performance on the most recent labels available. If there are no labels yet, record the proxy metrics and set a date for when labels will arrive. Open a retraining task only when performance falls beyond the level agreed with the customer, along with a labelled dataset and an evaluation comparing the new model with the old one.

Three mistakes are common. The first is treating every drift alert as an order to retrain. The second is leaving the tool's default thresholds unchanged. The third, and hardest to spot, is monitoring only inputs and missing concept drift, which never shows up on a distribution chart.

## Turn this procedure into an edge when job hunting

When reading FDE job ads, look for phrases such as "model monitoring", "post-deployment" or "own production outcomes". When you see them, ask directly in the interview who owns the model after go-live, and offer the procedure above as your answer.

On a CV, a line such as "built a drift runbook with thresholds calibrated on 12 months of historical data, reducing false alerts" says more than any certificate, provided the numbers in it are your own.

Next time a customer asks "do we need to retrain?", the best answer is neither "yes" nor "no". It is: "PSI is in the investigate zone, labels arrive this Thursday, and here is the performance level we agreed would trigger retraining."

**Try this week:**

- Take an important feature of a model in production and calculate PSI between the training data and the last 30 days of production data, using the code in this article
- Rerun the calculation on past months to see what PSI a normal month produces, then set your alert threshold from that number
- Ask the customer how long real labels take to arrive and who does the labelling, then record the answers in the runbook

## Sources

- [What Is Population Stability Index (PSI)?](https://arize.com/glossary/population-stability-index-psi/)

- [Data drift algorithm](https://docs-old.evidentlyai.com/reference/data-drift-algorithm)

- [Introduction to Vertex AI Model Monitoring](https://docs.cloud.google.com/vertex-ai/docs/model-monitoring/overview)

- [MLOps: Continuous delivery and automation pipelines in machine learning](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning)

- [What is concept drift in ML, and how to detect and address it](https://www.evidentlyai.com/ml-in-production/concept-drift)

- [Model optimization](https://developers.openai.com/api/docs/guides/model-optimization)
