Overfitting definition
Overfitting is a modeling error in which a machine learning model learns the noise and quirks of its training data instead of the general pattern. An overfit model scores very well on the data it was trained on but performs poorly on new, unseen data, which makes its predictions unreliable once it is deployed in production.
What causes overfitting?
Overfitting happens when a model has more capacity than the data can support. A decision tree grown to full depth can create a separate branch for nearly every training record, memorizing them rather than learning a rule. The same happens with large neural networks on small datasets, models with hundreds of features and few rows, noisy or inconsistent labels, and training that runs for too many epochs.
Data leakage produces a related problem. If a feature quietly contains the answer, such as a "refund issued" flag in a fraud model, the model looks brilliant in testing and collapses in production. Strictly speaking that is a data error, not overfitting, but the symptom is the same: excellent offline scores that do not survive contact with real, live data.
How to detect overfitting
The clearest sign is a large gap between training and validation performance. If a classifier is near perfect on training data and much weaker on validation data, it has memorized rather than learned. Plot learning curves of both metrics as training proceeds, and use k-fold cross-validation so the result does not depend on one lucky split. Always keep a final test set the team never tunes against.
Watch for warning signs beyond the metrics too. A model that leans heavily on an ID column, a timestamp or one oddly specific feature is often memorizing. SHAP or feature importance plots reveal this quickly, and a sanity check on a few hand-picked new examples can expose a model that only works on familiar data.
How to prevent overfitting
- Collect more representative data, or use data augmentation for images and audio.
- Choose a simpler model or limit capacity, such as maximum tree depth.
- Apply regularization: L1 or L2 penalties, and dropout in neural networks.
- Use early stopping when validation loss stops improving.
- Remove weak or leaky features through feature selection.
- Use ensembles such as random forests that average many models.
- Start from a pretrained model instead of training a large one from scratch.
- Tune hyperparameters with cross-validation, never against the final test set.
Overfitting vs underfitting
Underfitting is the opposite failure: the model is too simple to capture the real pattern, so it performs poorly on both training and validation data. A straight line fitted to clearly curved data underfits. The goal sits between the two, a balance usually described as the bias-variance trade-off. High bias means underfitting, high variance means overfitting, and tuning moves the model toward the point where validation error is lowest.
Large language models add a twist. They can memorize specific passages from training data and repeat them, which is a privacy and copyright concern as well as a generalization one. Evaluations for such models therefore check for verbatim recall of training text, not only accuracy on held-out benchmark questions.