Small cell lung cancer (SCLC) is one of the most aggressive forms of lung cancer, accounting for roughly 10-15% of all lung cancer cases but responsible for a disproportionately large share of lung cancer deaths. Its five-year survival rate is dramatically lower than other lung cancer types, particularly when detected at advanced stages with distant metastases.
Rapid spread to vital organs - including the brain, liver, bones, and adrenal glands - is a hallmark of SCLC. This extensive metastatic ability stems from the cancer's rapid cell proliferation, invasiveness, and resistance to programmed cell death (apoptosis), driven in part by abnormal expression of growth factors and receptors at the molecular level.
Early identification of metastatic stage at diagnosis is crucial because patients often miss the optimal treatment window before widespread spread occurs. Accurate staging directly determines whether more aggressive therapy is warranted and can meaningfully extend disease-free and overall survival.
Traditional statistical methods like logistic regression and Cox models have limitations in predicting SCLC metastasis - they struggle with high-dimensional data and are sensitive to sample size and variable selection biases. Machine learning offers greater predictive power but historically has faced a critical transparency problem: clinicians cannot see inside the decision-making process (the "black box" problem).
The SEER database (Surveillance, Epidemiology and End Results, maintained by the National Cancer Institute) served as the data source, covering SCLC patients diagnosed between 2004 and 2017. The study identified patients using ICD-O3 standard pathological codes covering small cell carcinoma in its various morphological subtypes.
12,128 patients were included in the final analysis after exclusion of records with missing key information, duplicate entries, and non-SCLC diagnoses. All patients had histopathologically confirmed SCLC, known TNM staging, and had received some form of treatment. Age and gender were not restricted, aiming for broad generalizability.
Metastatic staging was standardized across the study period into four categories: M0 (no distant metastases; 4,439 patients), M1a (limited distant metastases such as pleural involvement; 1,138 patients), M1b (single-organ distant metastasis; 6,054 patients), and M1c (multiple-organ metastases or high tumor burden; 497 patients).
Key demographic and clinical characteristics varied across stages in expected ways. Survival time dropped sharply with stage progression (20 months median for M0, 2 months for M1c). Tumor size was larger in higher stages, and advanced N stage (lymph node involvement) was more common in M1c patients. The cohort was predominantly White (83.9%) with roughly equal gender distribution.
Lasso regression was used first for feature selection across each of the four metastatic stage models, applying L1 regularization to automatically identify and retain only the most predictive variables while shrinking irrelevant ones to zero. Cross-validation optimized the regularization parameter for each stage model, ultimately identifying 10 variables consistently important across all four stages.
Ten machine learning algorithms were trained and compared, spanning a wide range of approaches: k-nearest neighbors (KNN), decision trees, random forests, XGBoost, support vector machines (SVM), LightGBM, multilayer perceptrons (neural networks), elastic net, ridge regression, and multinomial logistic regression. This diversity enabled a comprehensive head-to-head comparison.
Class imbalance was addressed using SMOTE (Synthetic Minority Oversampling Technique), which generates synthetic examples of underrepresented classes (especially M1c, with only 497 patients) within each cross-validation fold to improve model sensitivity to rare but clinically important cases. Models were evaluated using accuracy, F1-score, ROC-AUC, and precision-recall AUC (PR-AUC).
SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) were applied to the best-performing model to open the black box. SHAP uses game theory principles to quantify each feature's contribution to predictions across all patients, while LIME provides local explanations for individual cases - together making the model's reasoning transparent and clinically actionable.
K-nearest neighbors (KNN) consistently outperformed all other machine learning algorithms across both cross-validation on the training set and independent testing on the held-out test set. KNN achieved the highest and most stable PR-AUC and ROC-AUC values across all 10 cross-validation folds - indicating strong discrimination and reliable calibration across all four metastatic stages.
On the independent test set, KNN achieved accuracy of 0.806, balanced accuracy of 0.871, F1-score of 0.796, ROC-AUC of 0.894, and PR-AUC of 0.803 - the top scores across 12 evaluation metrics. These results demonstrate strong generalization to unseen data and robust performance even under class imbalance conditions.
Other nonlinear models performed competitively: XGBoost, SVM, and MLP all showed favorable performance, especially for rare M1c cases where KNN, XGBoost, and SVM showed better precision-recall performance than linear models. Linear approaches (Ridge, LASSO, Elastic Net) and decision trees consistently underperformed across most metrics.
Class-specific performance was particularly important for clinical utility. For M1c - the most advanced and rarest stage - KNN, XGBoost, and SVM achieved the best discrimination, though with some precision drop at high recall levels. Correctly identifying these high-burden cases is critical for directing patients toward the most aggressive treatment strategies early.
SHAP analysis revealed the most influential predictors across all metastatic stages, in order: N stage (lymph node involvement), tumor size, and age at diagnosis - followed by marital status, tumor laterality, systemic therapy sequence, chemotherapy receipt, and race/ethnicity. These variables collectively account for most of the model's discriminatory power.
The influence of each variable is stage-dependent. For M0 predictions, lower lymph node burden (N0-N1) and smaller tumor size were the strongest factors favoring absence of distant metastasis. For M1b and M1c, higher age, larger tumor size, and higher N stage all shifted predictions toward more advanced stages, while demographic and treatment variables had smaller mixed effects.
SHAP force plots for individual patients illustrated how each variable combines additively to produce a prediction for a specific case. For example, a patient with low N stage and small tumor size would strongly favor an M0 prediction, while a patient with high N involvement, large tumor, and older age would shift decisively toward M1b-M1c - matching clinicians' intuitions about disease biology.
Marital status appeared as a notable predictor - a sociological variable that may serve as a proxy for social support, health-seeking behavior, or access to care. Married patients comprised the majority across all stage groups, and its role in the model reflects broader patterns in cancer outcomes research linking social support to diagnosis timing and treatment adherence.
KNN's success in this setting reflects its non-parametric, instance-based nature. Rather than fitting explicit mathematical relationships between features and outcomes, KNN classifies patients based on the stage assignments of their most similar neighbors in the feature space. This approach adapts naturally to nonlinear decision boundaries and heterogeneous patient subgroups.
The model's interpretability was enhanced through SHAP and LIME analyses, which transform a complex classifier into a transparent decision-support tool. Healthcare professionals can see not only what the model predicts, but why - which specific patient characteristics (and in which direction) drove that prediction. This transparency is essential for clinical adoption.
The model performed well for rare cases. A sensitivity analysis excluding patients with prior primary malignancies (sequence number greater than 1) confirmed consistent performance, ruling out confounding from multiple cancer histories. Per-class metrics, calibration curves, and confusion matrices were all reported to give a complete picture of performance beyond summary AUC values.
Limitations include the US-centric SEER database, which may not fully represent global SCLC populations with different genetic, dietary, and environmental risk factors. Data entry errors and missing values in the registry could also affect model reliability. The model is intended for diagnostic-time staging rather than predicting future metastasis development over time.
Accurate metastatic stage prediction at diagnosis has direct therapeutic implications. For patients predicted to have advanced M1b or M1c disease, earlier consideration of intensified chemotherapy regimens or early palliative care planning could meaningfully affect both survival and quality of life. The model could serve as a clinical decision-support layer within existing diagnostic workflows.
The model supports personalized medicine in SCLC by enabling treatment stratification based on individual patient data rather than population averages. Patients at high predicted risk of multi-organ metastases could be flagged for more aggressive workup, earlier systemic therapy initiation, or enrollment in clinical trials targeting advanced-stage disease.
Integration into clinical workflows requires thoughtful implementation. The model needs access to the same variables collected during standard diagnostic evaluation (age, tumor size, T and N staging, treatment plan), and predictive outputs must be delivered in a timeframe that informs rather than delays clinical decision-making.
Future work should expand the dataset beyond the SEER registry to include diverse international populations, additional molecular biomarkers, and prospective longitudinal data enabling true prediction of future metastasis development (as opposed to classification of known staging at diagnosis). Incorporating emerging biomarker discoveries as they arise will be critical to keeping the model clinically relevant over time.
This study demonstrates that machine learning - specifically the KNN algorithm paired with SHAP interpretability analysis - can accurately classify SCLC metastatic stages from routinely available clinical data, outperforming both traditional statistical approaches and more complex tree-based ensemble methods.
The integration of model interpretability tools represents a significant step beyond typical machine learning studies in oncology. By making the model's reasoning transparent, the research directly addresses the primary barrier to clinical adoption: clinicians' legitimate need to understand why a model predicts what it predicts before acting on that prediction.
SHAP analysis confirmed that the model's key drivers (N stage, tumor size, age at diagnosis) align with established clinical understanding of SCLC biology, providing biological plausibility and reinforcing trust in the model's outputs rather than requiring clinicians to treat its predictions as unexplained oracle outputs.
Ultimately, the value of this work lies in its potential to reduce delays in appropriate treatment escalation for SCLC patients with occult or underestimated metastatic burden. By providing accurate, explainable predictions from standard diagnostic data, such tools could help ensure that patients receive appropriately aggressive or appropriately conservative treatment plans at the earliest possible moment.