Model Drift definition
Model drift is the decline in a machine learning model's accuracy after deployment because the data it sees in production, or the relationship between that data and the outcome, has changed since training. Its main forms are data drift and concept drift. Drift is detected through monitoring and usually corrected by retraining on recent data.
Why does model drift happen?
A model is a snapshot of the world at training time. When that world changes, the snapshot goes stale. Customer behavior shifts with seasons, prices and competitors. Fraudsters change tactics once their old ones are blocked. A new product line introduces items the model never saw. Even internal changes, such as a form that starts recording weight in grams instead of kilograms, can quietly break a model.
Drift is dangerous because it is silent. The model keeps returning predictions with the same apparent confidence, the API keeps responding and no error appears in the logs. Accuracy simply erodes until someone notices bad decisions downstream, sometimes weeks later and often only after a customer, auditor or finance team asks why the numbers stopped adding up.
Types of model drift
- Data drift (covariate shift): the distribution of inputs changes, such as a new age mix of customers.
- Concept drift: the relationship between inputs and outcome changes, such as what signals fraud.
- Label or prior drift: the base rate of the outcome changes, such as default rates rising in a downturn.
- Upstream data drift: a pipeline, schema or unit change upstream alters the inputs.
- Patterns over time: drift can be sudden, gradual or seasonal and recurring.
How to detect model drift
The best signal is real performance: compare predictions with actual outcomes as they arrive and track accuracy, precision or error over time. Outcomes are often delayed, though, since you only learn whether a loan defaults months later. Teams therefore also monitor input and prediction distributions, which can warn of trouble long before the true labels arrive.
- Statistical tests: Population Stability Index, Kolmogorov-Smirnov, Jensen-Shannon or Wasserstein distance per feature.
- Prediction monitoring: shifts in the share of positive predictions or average scores.
- Data quality checks: null rates, new categories and out-of-range values.
- Tools: Evidently, Arize, WhyLabs, Amazon SageMaker Model Monitor and Vertex AI Model Monitoring.
How to fix model drift
Start by ruling out a broken pipeline, because many apparent drift alerts are really data bugs. If the change is real, retrain on recent data, either on a fixed schedule or when monitoring crosses a threshold. Weighting recent records more heavily, or training only on a rolling window, helps the model follow gradual change. Every retrained model should pass the same evaluation gate before replacing the current one.
For sudden shocks, retraining may not be possible until enough new labeled data exists. In the meantime, tighten thresholds, send more cases to human review or fall back to simpler rules. Document each drift event so the team learns which features and segments are most fragile and can watch them more closely.
Example: drift in fraud detection
A payments company's fraud model performs well for months, then recall drops as attackers move to a merchant category the model rarely saw. Monitoring flags a rise in chargebacks and a shift in transaction mix. Analysts label the new pattern, the model is retrained and recall recovers. Nexzem sets up this kind of monitoring and retraining loop as part of every production ML system it delivers.