Artificial intelligence models are increasingly used to support medical decisions, from diagnosing cancer to predicting treatment outcomes. However, a fundamental tension exists in AI development: as models become more complex and accurate, they also become harder for humans to understand. This is often called the black-box problem.
In healthcare settings, this opacity is not just a technical inconvenience -- it has real consequences. Clinicians who cannot understand why an AI made a recommendation may be unable to evaluate whether to trust it, potentially leading to over-reliance on incorrect predictions or under-use of correct ones. Patients, too, may be less likely to follow treatment recommendations they don't understand.
Regulators and policy makers have taken notice. The U.S. National Institute of Standards and Technology (NIST) has emphasized that AI trustworthiness cannot be assumed from technical performance alone -- trust must come from the user's ability to understand the system. This has fueled the growing field of Explainable AI (XAI).
XAI approaches face their own challenges, including lack of expertise among users, potential for biased explanations, and the difficulty of producing explanations that are both accurate and understandable in complex clinical contexts. The need for strong collaboration between XAI systems and domain experts like physicians and radiologists is widely recognized.
This study demonstrates the use of two widely adopted explainability tools -- LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (Shapley Additive exPlanations) -- to explain individual predictions made by a deep neural network trained to classify cancer as benign or malignant.
The neural network was trained on the UCI Breast Cancer Wisconsin dataset, a well-known benchmark dataset containing 569 cases with 30 numerical features derived from digital images of cell nuclei in breast tissue biopsies. The dataset is widely used in machine learning research as a testbed for classification and interpretability methods.
While the application domain is breast cancer, the authors emphasize that the methodology -- combining a high-accuracy neural network with LIME and SHAP analysis -- is broadly applicable to any medical classification problem, including other cancer types or diagnostic tasks where neural network interpretability is needed.
The paper focuses not on maximizing prediction accuracy but on demonstrating how post-hoc explanation tools can reveal the reasoning behind neural network predictions in a way that is meaningful to medical professionals, increasing trust and enabling better human-AI collaboration.
LIME works by explaining a single prediction at a time, answering the question: which input features most influenced this particular classification? It does this by slightly perturbing the input data many times and observing how the model's output changes in response to each perturbation.
For each data point to be explained, LIME generates hundreds of slightly modified versions of that data point, runs all of them through the trained neural network, and then fits a much simpler linear model to describe the relationship between the perturbations and the model's outputs near that specific data point.
The result is a set of feature weights -- positive weights indicate features pushing the prediction toward one class (e.g., malignant), while negative weights indicate features pushing toward the other class (e.g., benign). These weights are local approximations valid only near the original data point, not global statements about the model.
LIME is described as model-agnostic because it does not look inside the neural network -- it treats the model as a black box and probes it from the outside. This means the same LIME approach can explain predictions from any type of model, from simple decision trees to complex deep neural networks.
SHAP takes a different approach, rooted in game theory. It treats the model prediction as a game and each input feature as a player, then asks: how much does each feature contribute to the final prediction relative to a baseline prediction that uses no features at all?
SHAP achieves this by evaluating the model for every possible combination of input features, measuring how much each combination changes the prediction relative to the baseline. These changes are combined using a mathematically principled averaging method called Shapley values, which ensures fair attribution across all possible feature combinations.
Unlike LIME, SHAP can produce both local explanations (for individual predictions) and global summaries (across an entire dataset). The global SHAP summary plot shows, for each feature, how its values across all patients relate to higher or lower predicted probabilities -- providing a bird's-eye view of model behavior.
SHAP also generates dependency plots that reveal interactions between features, showing how the influence of one feature on the prediction changes depending on the value of another feature. This can reveal clinically meaningful relationships that are not apparent from accuracy metrics alone.
The deep neural network used in the study consists of three hidden layers with ReLU activation functions and a softmax output layer. It takes 30 numerical features describing cell nucleus geometry and texture as input and outputs the probability of a mass being benign or malignant.
The network was trained on 80% of the 569-case dataset and evaluated on the remaining 20% as the test set. It achieved a test accuracy of 97% -- placing it among the highest-performing models published for this dataset, where the state-of-the-art benchmark is approximately 98.6%.
The 30 input features include measurements such as radius, texture, perimeter, area, smoothness, compactness, concavity, concave points, symmetry, and fractal dimension of cell nuclei. For each of these 10 properties, three statistical summaries are included: the mean, standard deviation, and the mean of the three largest values (called the worst value).
This high accuracy demonstrates that neural networks are effective at identifying malignant masses from cell nucleus measurements. However, accuracy alone does not explain why specific patients were classified as they were -- the primary motivation for applying LIME and SHAP to the trained model.
For malignant cases, both LIME and SHAP consistently identified features such as texture (worst), concave points (worst), and smoothness (worst) as the most important drivers of the malignant prediction. These features represent measurements at the most extreme cell nucleus values in a tissue sample.
For benign cases, LIME highlighted that lower values of features like radius (worst), concavity (mean), and perimeter (worst) drove predictions toward benign classifications. This pattern -- smaller and less irregular nuclei being associated with benign masses -- is consistent with established medical understanding of cancer cell morphology.
The SHAP global summary plot revealed that the worst-case values of features (representing the most extreme measurements in a sample) are consistently more important for distinguishing benign from malignant masses than the mean values. This finding aligns with pathological understanding that cancer cells are particularly characterized by their most atypical-appearing nuclei.
Importantly, both LIME and SHAP produced consistent insights despite using entirely different mathematical approaches. This convergence between the two tools provides stronger evidence that the identified features are genuinely important to the model's decision-making, rather than being artifacts of any single explanation method.
SHAP dependency plots allow researchers to visualize how the influence of one feature on predictions changes depending on another feature's values. For example, the texture (mean) of cell nuclei showed a linear decrease in its contribution to benign predictions as its value increased -- meaning higher texture values push the model toward malignant predictions.
The same dependency plot revealed that texture (worst) tends to increase together with texture (mean), suggesting these two features are correlated in the dataset. This kind of feature interaction information is invisible in standard accuracy metrics and would not be apparent from the model's architecture alone.
For concave points (mean), the dependency plot showed a similar pattern where higher values decreased the probability of a benign prediction. However, the relationship between concave points (mean) and its highest-interacting feature -- symmetry (standard deviation) -- was not linear, showing that different patients with the same concavity values can have very different symmetry characteristics.
These interaction insights have clinical relevance: they suggest that malignant cell nuclei are not only larger and more irregular, but that multiple geometric properties tend to be simultaneously extreme in malignant cases. This multi-feature co-variation may reflect underlying biological properties of cancer cell growth and division.
This study demonstrates that explainability tools can transform opaque neural network predictions into actionable insights for medical practitioners. By revealing which features drove each individual prediction, LIME and SHAP enable clinicians to verify that AI recommendations are based on medically meaningful patterns rather than spurious correlations.
Both tools confirmed predictions that align with established medical knowledge about cancer cell morphology, which increases confidence that the model is learning genuine biological signals. When LIME and SHAP agree with each other and with domain knowledge, this convergence provides a strong validation that the AI is behaving as intended.
However, there are practical limitations. Both LIME and SHAP require running the trained model many times to generate explanations, making them computationally intensive and unsuitable for real-time applications. They are best suited to post-hoc analysis -- understanding why a model reached a particular conclusion after the prediction has already been made.
The authors suggest a promising future direction: using the internal abstract features learned by the neural network's hidden layers as inputs to LIME and SHAP, rather than the raw input features. This could provide even deeper insights into what the model has learned and may also benefit applications like transfer learning, where knowledge learned for one medical task is applied to another.