XAI definition
Explainable AI (XAI) is a set of methods that make the predictions of machine learning models understandable to people. XAI shows which inputs influenced a decision and how much, using techniques such as SHAP, LIME and feature importance, so developers, regulators and affected users can check, trust and challenge automated decisions.
Why explainable AI matters
When a model decides who gets a loan, which claim is investigated or which patient is flagged, people need to know why. Regulation increasingly demands it. In the US, lenders must give applicants specific reasons for adverse credit decisions. In Europe, GDPR gives individuals rights around solely automated decisions, and the EU AI Act sets transparency and documentation duties for high-risk systems.
Explanations also make models better. A well-known study on the LIME technique showed an image classifier separating wolves from huskies by detecting snow in the background rather than the animal. Explanations exposed the shortcut. The same kind of check regularly catches leaky features, proxy variables for protected attributes and spurious correlations in business models before they cause harm.
Explainable AI techniques
- Interpretable models: linear models, small decision trees and explainable boosting machines (InterpretML).
- SHAP: assigns each feature a contribution to a prediction, based on Shapley values from game theory.
- LIME: fits a simple local model around one prediction to approximate the complex model.
- Permutation importance and partial dependence plots: show global feature effects.
- Counterfactual explanations: "the loan would be approved if income were higher by this amount".
- Saliency maps and Grad-CAM: highlight image regions that drove a vision model's output.
- Token attribution methods: partial insight into transformer models, with well-known limits.
Global vs local explanations
Global explanations describe how a model behaves overall, such as "payment history is the strongest driver of risk scores". They help data scientists, risk teams and regulators judge whether the model is sensible. Local explanations describe one decision, such as "this application scored high risk mainly because of two recent missed payments". They are what customers, agents and appeal reviewers need. A complete XAI setup provides both.
Limitations of explainable AI
Post-hoc explanations are approximations of the model, not the model itself. SHAP and LIME can produce unstable or misleading results when features are strongly correlated, and a persuasive chart can create false confidence. For high-stakes decisions, some researchers argue for inherently interpretable models instead, and on structured data they often lose little accuracy compared with black-box models.
Large language models are harder still. A model's written reasoning is not guaranteed to reflect how it actually reached an answer. For LLM features, practical transparency usually means citing retrieved sources, logging prompts and tool calls, and evaluating outputs systematically, rather than claiming a full explanation of the network's internal computation.
How to add explainability to a model
Decide who needs an explanation and what they will do with it before choosing a method. A credit model might return the top three SHAP-based reason codes with every score, translated into plain language for the applicant and stored for audit. Test explanations with real users, since a technically correct chart can still confuse the person reading it. Nexzem builds explanations into the API response and the review screen, so they are available at the moment a decision is made.