Machine learning (ML) models are increasingly being used in oncology research -- predicting who will respond to treatment, which imaging features indicate cancer, and how long patients are likely to survive. These models often outperform traditional statistical methods in accuracy. But there is a fundamental problem: as models become more powerful, they become harder to understand.
Traditional statistical tools like logistic regression produce odds ratios and p-values that oncologists can interpret intuitively. A complex ML model like a neural network or gradient-boosted tree may make more accurate predictions, but its internal workings resemble a black box -- inputs go in and predictions come out, with little insight into why a specific prediction was made.
This opacity is a major barrier to clinical adoption. Clinicians, patients, hospital administrators, and regulators all want to know not just that a model predicts something correctly, but how it arrived at that prediction and which factors drove it. A physician who cannot explain to a patient why an AI recommends a particular treatment is unlikely to use that AI.
Explainable artificial intelligence (XAI) is a set of techniques specifically designed to open the black box -- producing visualizations, feature importance scores, and interpretable summaries that reveal the inner logic of complex ML models. This review surveys how two leading XAI frameworks, SHAP and LIME, are being applied across oncology.
SHAP (SHapley Additive exPlanations) is based on game theory. The idea is to calculate the fair contribution of each player (feature) to the final outcome (prediction) by averaging that feature's contribution across all possible combinations of features. The result is a SHAP value for each feature and each prediction -- telling you exactly how much that feature pushed the prediction up or down.
SHAP can be applied at two levels: to understand individual predictions (why did the model give this specific patient a high cancer risk score?) and to understand the global model behavior (across all patients, which features matter most and how do they interact?). SHAP is particularly efficient for tree-based models like XGBoost and gradient-boosted trees.
LIME (Local Interpretable Model-agnostic Explanations) works differently. It focuses on explaining single predictions by creating a simplified local model around that specific data point. LIME perturbs the input data slightly and fits a simple linear model to the results, revealing which features are most important in the local neighborhood of that prediction. LIME is computationally efficient and works well for explaining individual cases.
Both frameworks are model-agnostic, meaning they can be applied to any ML model regardless of its internal architecture -- neural networks, random forests, support vector machines, or any other algorithm. This universality makes them broadly useful across all ML applications in oncology.
Several prostate cancer studies illustrate XAI's clinical value. Li et al. used SHAP with an extreme gradient-boosted model to explore how PSA, percent positive biopsy cores (PPCs), and Gleason score interact to influence survival. Standard regression could not have captured these nonlinear interactions.
SHAP interaction plots revealed that PPCs are largely irrelevant for survival prediction when Gleason score is 7 or below, but become important at higher Gleason scores -- and the critical PPC threshold at high Gleason score is 70%, not the 50% currently used in clinical risk stratification. This type of nuance, invisible to linear models, could refine how patients are classified and treated.
For pre-biopsy prostate cancer diagnosis, Suh et al. used SHAP to identify the most important predictive features for clinically significant prostate cancer. The analysis identified known predictors (PSA, Gleason score) and showed how changing each factor individually influenced risk -- enabling the team to build an accessible online risk calculator grounded in the ML model's logic.
Kwong et al. used SHAP to predict extraprostatic extension (whether cancer has spread beyond the prostate capsule) before surgery. The dependence plots revealed that percent positive cores were not meaningful predictors until reaching approximately 75% -- information that can directly guide surgical planning by helping surgeons know when nerve-sparing approaches are too risky.
Radiomics -- extracting quantitative features from medical images -- is a field often criticized for being opaque. A radiomic model might extract thousands of texture, shape, and intensity features from a tumor image and combine them into a risk score, but without knowing which features matter most, the result is untrustworthy.
XAI addresses this directly. In brain tumor studies, SHAP was applied to deep learning models identifying tumor subtypes from MRI, producing color-coded overlays where red pixels increase a classification probability and blue pixels decrease it. Radiologists and oncologists can see exactly which parts of the image the model is responding to -- and verify whether those regions align with clinical knowledge.
For feature selection in model building, SHAP has been used to eliminate low-importance features before finalizing models. In one glioma study, removing features with low average SHAP values improved the model's AUC from 0.710 to 0.753 -- demonstrating that XAI doesn't just explain models, it actively improves them by identifying redundant or uninformative variables.
LIME has been applied to classify breast cancer lymph node metastases from ultrasound images, creating highlighted image overlays that show which regions drove the classification. This allows pathologists to cross-check AI decisions against their own visual assessment, building trust and catching errors before they reach patients.
One of the most clinically impactful uses of XAI is identifying treatment selection thresholds -- specific patient characteristics above or below which a particular treatment offers benefit versus harm. These thresholds are often nonlinear and context-dependent, making them nearly impossible to identify with standard regression.
In endometrial cancer, SHAP plots from an ML model identified that patients with four or more positive lymph nodes benefited most from adjuvant chemoradiation -- while those with fewer nodes did equally well with chemotherapy alone and potentially worse with radiation alone. This addresses ongoing clinical controversy about optimal adjuvant treatment sequencing after major trials.
In radiation treatment planning, XAI has been applied to quality assurance workflows. ML models trained on historical treatment plans can predict whether a new radiation plan will meet dose objectives. SHAP identifies which dosimetric parameters drove the prediction, giving physicists and radiation oncologists actionable feedback on how to improve plans that don't meet standards.
Chen et al. applied SHAP and LIME specifically to prostate cancer radiation quality assurance, explaining ML models that evaluate whether treatment plans can be safely delivered. This kind of interpretable AI can streamline clinical workflows, reduce errors, and help train less experienced staff by making the basis of quality decisions transparent.
The review identifies five distinct categories of value that XAI adds to oncology ML models: (1) delineating global feature importance -- which variables matter most across all patients; (2) characterizing individual prediction importance -- explaining why a specific patient received a specific prediction; (3) visualizing nonlinear relationships -- showing how variables interact in ways that linear models cannot capture.
(4) Identifying prognostic and predictive thresholds -- finding the specific values of variables where clinical impact changes, such as a PSA level above which risk increases sharply, or a lymph node count above which a specific treatment becomes beneficial. (5) Feature selection and construction -- using SHAP importance scores to identify the most informative features and remove redundant ones, improving model efficiency and reducing overfitting.
Across all reviewed studies, SHAP was the most commonly used framework, reflecting its versatility in explaining both individual predictions and global model behavior. LIME was preferred in applications focused on individual patients -- particularly in pathology and imaging -- where case-by-case explanation is more important than population-level trends.
The authors note that SHAP and LIME produce approximations, not exact explanations. Some computational simplifications are necessary to make explanations feasible. This means XAI outputs should be interpreted as probabilistic guides, not definitive truths -- and models should be rigorously validated before XAI outputs are used to make clinical decisions.
A key limitation is that explanations are only as good as the models they explain. If a model has been trained on biased or unrepresentative data, XAI will faithfully explain those biases rather than correct them. Robust model validation must always precede XAI application in clinical settings.
Most reviewed studies lack external validation -- testing models on patient data from different institutions. A model that performs well at one cancer center may not generalize to another due to differences in patient populations, imaging protocols, or clinical practices. This limits the immediate clinical applicability of most published XAI-oncology studies.
The authors note that for simpler problems where accurate, inherently interpretable models (like LASSO regression or decision trees) can achieve comparable performance, using a complex ML model plus XAI may be unnecessary overhead. Model selection should consider both predictive performance and inherent interpretability before reaching for black-box algorithms plus post-hoc XAI.
Despite these limitations, XAI represents a necessary bridge between the growing power of ML in oncology and the interpretability requirements of clinical medicine. As electronic health records grow richer with genomic, imaging, and clinical data, explainable ML models that can process this complexity while remaining accountable will be increasingly central to precision oncology.
SHAP and LIME are now established frameworks for making oncology ML models interpretable. Their use across prognostication, diagnosis, radiomics, pathology, treatment selection, radiation therapy, and epidemiology demonstrates that XAI has broad applicability throughout the cancer care continuum.
In prostate cancer specifically, XAI has already delivered clinically meaningful insights: identifying unexpected risk stratification thresholds, helping design biopsy decision tools, predicting surgical risk of extraprostatic extension, and explaining radiation quality assurance decisions.
The fundamental challenge in clinical AI -- that more accurate models tend to be less interpretable -- is not fully solved by XAI, but meaningfully addressed. By converting complex model outputs into intuitive charts, color-coded images, and ranked feature lists, XAI makes powerful ML accessible to oncologists who are not data scientists.
Future oncology AI research should routinely incorporate XAI frameworks alongside model development. This is not merely a technical add-on -- it is a prerequisite for responsible deployment of AI in clinical settings where predictions directly affect patient care decisions.