Study Purpose This study developed a cost-efficient, explainable AI diagnostic tool for lung cancer using routine clinical data -- blood tests, symptoms, and patient history -- rather than expensive imaging. The goal was to create a tool accessible to resource-limited settings.
Dataset The study used data from 720 patients collected in India, with 74 clinical features including complete blood counts, inflammatory markers, symptoms like chronic cough and dyspnea, smoking history, and occupational exposure details.
Model Performance A stacking ensemble model combining an Artificial Neural Network (ANN) and Deep Neural Network (DNN) achieved an AUC of 0.86 and accuracy of 85.58%, surpassing all individual base models tested.
Explainability SHAP (SHapley Additive exPlanations) analysis identified the Erythrocyte Sedimentation Rate (ESR) as the single most important feature, followed by hemoglobin, platelet count, and symptom-based features.
Feature Categories The 74 clinical features spanned hematological parameters (CBC, ESR, LFTs), symptoms (cough duration, hemoptysis, chest pain, weight loss), risk factors (smoking pack-years, occupational exposures to asbestos, radon), and basic demographics.
Population Context The Indian dataset is significant because India has a high burden of lung cancer with unique risk factor profiles, including high rates of non-smoking-related lung cancer linked to indoor air pollution and occupational exposures, which differ from Western populations.
Cost-Efficiency Rationale In India and many developing countries, CT scans and PET imaging remain expensive and not universally accessible. A diagnostic tool based on blood tests and symptom questionnaires could extend early detection to primary care settings and rural hospitals.
Class Imbalance Challenge Among the 720 patients, lung cancer cases were a minority, creating a class imbalance problem where standard models tend to predict the majority (non-cancer) class too often, missing actual cancer cases.
ADASYN Algorithm Adaptive Synthetic Sampling (ADASYN) was used to address class imbalance. Unlike simple oversampling, ADASYN generates more synthetic samples for minority class instances that are harder to classify, focusing data augmentation where the model most needs help.
Impact on Rare Case Detection Without ADASYN, the model identified only 11.4% of lung cancer cases in the minority class correctly. After applying ADASYN to the training data, detection of rare cancer cases improved dramatically to 62.9%.
Why This Matters In a screening or diagnostic context, missing cancer cases (false negatives) is clinically more harmful than false positives. ADASYN specifically improved recall for cancer cases, reducing the rate of missed diagnoses.
Synthetic Sample Quality ADASYN generates synthetic samples by interpolating between real minority-class examples in feature space. The quality of these synthetic patients was validated by checking that model behavior on synthetic data matched trends from real data.
Stacking Framework The stacking ensemble uses base model predictions as features for a meta-learner. An ANN (Artificial Neural Network with 2-3 hidden layers) and DNN (Deep Neural Network with 5+ layers) served as base models, with a logistic regression meta-learner combining their outputs.
Why Stacking Works Each base model captures different aspects of the data. The ANN captures relatively simple non-linear patterns while the DNN captures complex high-order interactions between clinical features. Stacking combines these complementary strengths.
Comparison to Individual Models The stacking ensemble (AUC 0.86) outperformed ANN alone (AUC approximately 0.80), DNN alone (AUC approximately 0.82), random forest (AUC approximately 0.79), and traditional logistic regression (AUC approximately 0.74).
Hyperparameter Optimization Grid search cross-validation was used to tune network depths, learning rates, regularization, and dropout rates for both ANN and DNN components, ensuring the ensemble was optimally configured.
Top Predictors SHAP analysis ranked features by their average absolute contribution to predictions. ESR (erythrocyte sedimentation rate -- a blood inflammation marker) was the top feature, consistent with the systemic inflammation that lung cancer induces.
Other Key Features Following ESR, hemoglobin level, platelet count, neutrophil-to-lymphocyte ratio, cough duration, and smoking pack-years were among the most influential features. These collectively capture inflammation, hematological effects, and direct cancer risk factors.
Individual Prediction Explanations For each patient, SHAP produces a personalized feature contribution plot showing which specific values pushed the prediction toward or away from cancer. This transparency is critical for clinicians who need to understand why the AI flagged a patient.
Clinical Validation of SHAP Findings The features identified by SHAP align with established clinical knowledge about lung cancer biology, providing confidence that the model is learning genuine disease patterns rather than spurious statistical correlations.
Geographic Specificity The model was trained on Indian patients with unique lung cancer risk profiles. Performance may differ in Western populations where smoking-related lung cancer predominates and different genetic and environmental backgrounds are present.
Lack of Imaging Integration The study intentionally excluded imaging to maintain cost-efficiency, but adding basic chest X-ray features or AI analysis of portable CXR could potentially boost performance without prohibitive cost.
Prospective Validation All model development and testing was retrospective. Prospective real-world deployment in primary care clinics, where the model would triage patients for further workup, is needed to assess actual clinical utility.
Longitudinal Applications Future work could explore whether the same feature set -- applied at multiple time points -- could detect rising cancer risk before symptoms become overt, enabling truly early detection rather than diagnosis of symptomatic disease.