Evidently AI: run drift checks before the client sees the data has changed

Three months after you leave the client, the scoring model is still running and reporting no errors, but the input data has changed. The model does not notice. Usually it is the business team that finds out, when the numbers start to look odd.

Evidently exists to fill exactly this gap. It is an open-source Python library for evaluating, testing and monitoring ML and LLM systems, released under the Apache 2.0 licence.

For an FDE, that licence has very practical consequences. An Apache 2.0 library installed directly in the client’s environment usually clears legal review more easily than a new service that needs a contract. You also leave behind a tool the client’s team can run themselves once you are gone.

Every measurement starts from a baseline

The package ships with more than 100 metrics, ranging from data drift detection to LLM judges, along with a declarative testing API. Tabular data quality and data drift checks are among the main use cases listed in the documentation.

The core logic is simple: compare current production data with a reference dataset. Oscar Wong, an MLOps engineer at Fuzzy Labs, describes the reference dataset as the benchmark to compare against. The blog post announcing Evidently’s version 0.0.1 in 2020 recommended choosing data from a period when the model was performing stably.

So the first question at the client is not “how do we install the library?” but “which month was the model working the way you wanted?”. If the baseline is off, every alert downstream is off too.

The drift threshold depends on how much reference data you have

Evidently does not use one drift measure for every case. The default method is chosen by the size of the reference dataset, and this is the point most likely to cause confusion at handover.

If the reference dataset has 1000 observations or fewer, Evidently uses statistical tests such as KS, chi-squared or Z-score. The drift score is a p-value and the default confidence level is 0.95, so any column with p ≤ 0.05 is treated as drifted.

When the reference dataset is larger, the tool switches to distance metrics such as Wasserstein or Jensen-Shannon, with a default threshold of 0.1.

Take an example. You hand over a loan-approval model with 12 features, and the reference dataset is 800 rows of validation data. Because 800 does not exceed 1000, each column is checked with a statistical test.

Suppose the reference dataset is later replaced with 5000 rows from the previous quarter. The measure has now switched to distance with a 0.1 threshold, and the same production batch can produce a different result. If the documentation does not record this, whoever takes over will not understand why alerts suddenly become more or less frequent.

Dataset-level drift is not a fixed default either; you declare it. The documentation’s example treats the whole dataset as drifted when 50% of columns drift. With 12 features, under that rule just 6 drifted columns flag the whole dataset.

Ask the client whether those 6 columns matter equally. Drift in an income column may be more worrying than five minor columns combined.

From report to checkpoint

A report is only useful if someone opens it. The next step, then, should be the declarative testing API: you write down pass/fail conditions, and drift becomes a step that can fail in CI, like a unit test.

The code below is a minimal sketch for the 12-feature loan model above.

# SKETCH, NOT VERIFIED AGAINST A SPECIFIC VERSION:
# class names (Report, Dataset, DataDefinition, DriftedColumnsCount, lt)
# and the result structure (result["tests"], "status") may differ between
# Evidently versions. Check the documentation for the version you install
# before putting this into a real pipeline.
import sys
import pandas as pd
from evidently import Report, Dataset, DataDefinition
from evidently.presets import DataDriftPreset
from evidently.metrics import DriftedColumnsCount
from evidently.tests import lt

reference = pd.read_parquet("validation_800_rows.parquet")
current = pd.read_parquet("production_last_batch.parquet")

definition = DataDefinition()
ref = Dataset.from_pandas(reference, data_definition=definition)
cur = Dataset.from_pandas(current, data_definition=definition)

# 1. Report: measure and summarise only, export HTML for the business team
report = Report([DataDriftPreset()])
report.run(current_data=cur, reference_data=ref).save_html("drift_report.html")

# 2. Test: same measurement, plus a pass/fail condition
#    lt(6): fail when 6 of 12 columns (50%) or more have drifted
suite = Report([DriftedColumnsCount(tests=[lt(6)])])
result = suite.run(current_data=cur, reference_data=ref).dict()

# 3. CI gate: stop the pipeline if any test does not pass
failed = [t for t in result["tests"] if t["status"] != "SUCCESS"]
if failed:
    print(failed)
    sys.exit(1)

The report and the test read the same data, but each has its own audience. The HTML file is for the business team to read. A non-zero exit code stops CI and raises the alarm, even when nobody remembers to open the report.

Evidently’s monitoring documentation suggests running batch checks on a schedule, hourly or daily, or triggering them when new data or labels arrive. For the loan model, running the check on every new batch of applications is how you catch drift before the client sees it in the results.

What the tool will not do for you

Drift tells you the data distribution has changed, not that the model is predicting wrongly. A feature can shift while prediction quality holds up, and the reverse can happen too. So do not promise the client that “no drift” means “the model is still good”.

The 0.95 and 0.1 thresholds are only defaults, not truths. Treat them as a starting point for agreeing with the business team which level of change needs a person to step in.

What to learn first?

Start with a single drift check on a model you understand well. Read the drift methods section of the documentation until you can explain the 1000-observation rule yourself. Only then add pass/fail conditions and put them into CI, as in the code above.

Leave the LLM side for later. When you need it, Evidently’s Tracely library collects data from running applications, and the documentation regards tracing as the best fit for LLM-powered applications.

How do you make this skill show when job hunting?

When reading FDE job ads, watch for phrases such as post-deployment model monitoring, data quality, regression testing or running LLM systems in production. They signal that the company needs someone who stays with the system after the demo, not just someone who builds prototypes.

The best proof is a small public repo. Take an open dataset, slice off part of it as the reference, deliberately skew a few columns in the rest, and let GitHub Actions run the check and fail at the right moment. The README should explain why you chose that threshold and who receives the alert.

On a CV, a line such as “set up drift checks in CI for a production model, with threshold documentation for the client team” carries far more weight than a list of tool names. If you have real figures from the project, such as the number of features monitored or how often the check runs, include them.

In interviews, prepare a story about a time a test failed: what signal raised the alarm, what you checked to tell harmless drift from a model going wrong, and how you agreed new thresholds with the business team.

A good handover does not mean the model never breaks. It means that when the data changes, the client’s team knows before its users do.