Single-modality models miss the full picture. Current AI approaches for lung cancer diagnosis typically rely on one type of data - either CT images, or genomic data, or clinical information alone. But lung cancer is a highly heterogeneous disease, with subtypes that differ in genetics, histology, and radiology. No single data type captures the full complexity of tumor biology, leaving single-modality models with fundamental accuracy limits.
The problem with standard segmentation. U-Net is one of the most widely used AI architectures for identifying tumor boundaries in medical images. However, its traditional single-pass approach extracts features only once, potentially missing subtle or complex tumor details - especially when different image types like CT scans and histopathology slides are involved. More iterative refinement is needed to handle the full complexity of lung tumor borders.
Survival prediction needs to handle nonlinearity. The classical Cox proportional hazards model - the standard statistical tool for survival analysis - assumes that each patient variable affects survival risk in a simple linear way. In reality, the interactions between clinical, imaging, and genomic factors are highly nonlinear and complex. Deep learning extensions of the Cox model can capture these interactions but require multimodal data integration to do so effectively.
Clinical models need to update over time. Once deployed, clinical AI models face a fundamental challenge: patient populations and medical knowledge evolve continuously. Static models trained at one point in time gradually become outdated as new patient data arrives. This study addresses this challenge by incorporating continuous learning mechanisms that allow the model to update from new patients without losing what it learned previously.
Iterative U-Net for tumor segmentation. The framework begins with a U-Net architecture - an encoder-decoder model with skip connections that preserves both high-level context and fine spatial detail. The key innovation is iterative refinement: rather than processing images once, the model performs multiple segmentation passes (typically 5 to 6 iterations), refining its prediction each time by incorporating the errors from the previous pass. This iterative process is especially beneficial for complex, irregular tumor boundaries and increased training time by 18% while improving segmentation precision.
Multimodal feature fusion for subtype classification. After segmentation, the framework extracts features from three data types simultaneously: CT images and histopathology slides are processed by a Convolutional Neural Network to capture spatial and morphological patterns, while genomic data including gene mutation profiles and RNA-Seq data is processed by a Multilayer Perceptron. The resulting feature vectors are concatenated into a unified multimodal representation that captures both the visual appearance and the molecular biology of the tumor.
Cross-attention for intelligent feature weighting. A cross-attention network dynamically calculates how much each data modality should influence the final prediction for each individual patient. For a patient where imaging features are highly informative, the model automatically assigns more weight to imaging features; for another patient where genomic markers are more predictive, it shifts weight toward genomics. This patient-specific weighting maximizes the information extracted from each available data source.
DeepSurv with elastic weight consolidation. For survival prediction, the framework uses DeepSurv - a deep learning extension of the Cox proportional hazards model that captures nonlinear relationships between patient features and survival risk. DeepSurv is combined with Elastic Weight Consolidation (EWC), a technique that allows the model to continuously learn from new patient data without forgetting its previously learned knowledge. EWC identifies which model parameters are most critical for prior predictions and prevents them from being overwritten by new training updates.
Three publicly available data types integrated. The framework was trained and tested using data from two major public repositories: The Cancer Imaging Archive provided 1,000 CT scan images and 500 histopathology images annotated by expert radiologists and pathologists, covering adenocarcinoma, squamous cell carcinoma, and large cell carcinoma subtypes. The Cancer Genome Atlas provided genomic profiles including gene mutation data, RNA-Seq expression, and clinical metadata for approximately 400 lung cancer patients.
Stratified cross-validation approach. The integrated dataset was divided into 70% for training, 15% for validation, and 15% for testing, with stratification to maintain equal proportions of each lung cancer subtype in all splits. Five-fold cross-validation was used to assess generalizability, averaging segmentation, classification, and survival prediction metrics across all folds to reduce the impact of data variability.
Bayesian optimization for hyperparameter tuning. Rather than manual trial-and-error or exhaustive grid search, the framework used Bayesian optimization to systematically identify the best combination of learning rates, dropout rates, number of neural network layers, and other model settings. This approach uses a probabilistic model (Gaussian process) to predict which hyperparameter configurations are most likely to improve performance, reducing the number of required experiments while finding better solutions.
SHAP for interpretability. SHapley Additive exPlanations were calculated for all predictions, assigning each input feature - including tumor size, EGFR mutation status, patient age, and comorbidities - a contribution score for every individual prediction. This makes the model transparent: clinicians can see which specific features drove each patient's subtype classification or survival risk estimate, rather than receiving a black-box output they cannot interrogate.
Superior tumor segmentation. The iterative U-Net achieved an average Dice Similarity Coefficient of 0.926 across CT and histopathology segmentation tasks - meaning it correctly identified 92.6% of the tumor area on average. This outperformed the three comparison methods, which achieved average Dice scores of 0.891, 0.874, and 0.852 respectively. Individual patient results ranged from 0.890 to 0.963, with larger and more irregular tumors requiring more iterations to converge.
88.8% classification accuracy with multimodal fusion. For predicting lung cancer subtype (adenocarcinoma, squamous cell carcinoma, or large cell carcinoma), the proposed model achieved 88.8% overall accuracy, with AUC-ROC of 0.915. Adenocarcinoma was classified most accurately (91.2%), while large cell carcinoma - the rarest subtype with the most overlap with squamous features - showed somewhat lower accuracy (86.7%). The 6% misclassification between large cell and squamous subtypes was traced to shared imaging characteristics, which genomic features helped resolve.
C-index of 0.795 for survival prediction. DeepSurv with full multimodal input (imaging, genomic, and clinical data) achieved a concordance index of 0.795 - meaning the model correctly ranked 79.5% of patient pairs by their relative survival risk. This outperformed comparison models ranging from 0.737 to 0.768. Adding each data modality incrementally improved performance: clinical data alone yielded 0.722, imaging and clinical combined reached 0.759, and the full multimodal combination achieved 0.795.
82.3% accuracy in three-tier risk stratification. When classifying patients into high-risk, medium-risk, and low-risk survival groups, the proposed model achieved 82.3% accuracy compared to 73.9 to 77.6% for comparison methods. Continuous learning with EWC allowed performance to improve by approximately 1 to 1.5% as new patient data was incrementally added in batches of 50, without degrading performance on previously learned cases.
Tumor size and EGFR mutations are the top predictors. SHAP analysis revealed that tumor size and EGFR mutation status were the two most influential features for both subtype classification and survival risk prediction, receiving the highest importance scores across the patient cohort. This aligns with established clinical knowledge: larger tumors indicate more advanced disease, while EGFR mutations define a molecularly distinct subtype with specific treatment options and prognosis patterns.
Cross-attention shifts weight by patient. Analysis of cross-attention weights showed that imaging features dominated for patients with large, morphologically distinctive tumors (attention weight up to 0.68 for imaging in high-risk cases), while genomic features carried more weight for patients with smaller or atypical lesions where visual features alone were ambiguous. This dynamic weighting makes the model more flexible and accurate across the heterogeneous spectrum of lung cancer presentations.
SHAP enables clinical dialogue. The SHAP explanation score for each patient prediction ranged from 0.65 for low-risk patients to 0.95 for the most complex high-risk cases - reflecting how clearly the model's reasoning is grounded in interpretable features. High SHAP scores for EGFR mutations and tumor irregularity suggest aggressive biology and could help oncologists prioritize targeted therapy or combination approaches for specific patients.
Overall interpretability score leads competitors. The proposed framework achieved an overall SHAP interpretability score of 0.736 compared to 0.634 to 0.682 for comparison methods, confirming that multimodal fusion with cross-attention not only improves accuracy but also produces more interpretable predictions - an important property for clinical trust and adoption.
Near-real-time processing speed. Processing one complete patient case - including CT segmentation, histopathology analysis, genomic feature extraction, subtype classification, and survival prediction - required only 4.2 seconds. This near-real-time performance, combined with peak GPU memory utilization of 70% during cross-attention fusion, makes the framework practically deployable on modern clinical computing hardware.
Continuous learning without forgetting. When new patient batches of 50 cases were added to simulate ongoing clinical deployment, the EWC mechanism updated the model's survival predictions while preserving knowledge from previous training. Hazard ratios for existing patients remained stable (changing by less than 0.02 in most cases) while model performance improved by approximately 1 to 1.5% - demonstrating that the framework can grow more accurate over time in a clinical setting without requiring complete retraining.
Supporting clinical decision workflows. The framework generates four outputs per patient: tumor segmentation maps for treatment planning, subtype classification with probability scores, three-tier survival risk group assignment, and SHAP feature importance breakdowns. This multi-output design aligns with clinical decision-making: surgeons use segmentation maps, oncologists use subtype classification for drug selection, and palliative care teams use survival risk stratification for treatment intensity planning.
Bridging computational and clinical reasoning. By correlating high SHAP scores with clinical factors familiar to oncologists - such as EGFR mutation status, tumor size, and comorbidities - the framework enables radiologists, pathologists, and oncologists to collectively interpret and act on AI predictions. Rather than replacing clinical judgment, the system serves as a data-driven decision support tool that surfaces and quantifies information that would otherwise require laborious manual synthesis across multiple data sources.
A unified, adaptive framework. The proposed multimodal deep learning framework successfully integrates tumor segmentation, subtype classification, and survival prediction into a single end-to-end system. By combining iterative U-Net segmentation, CNN-MLP multimodal fusion, cross-attention weighting, and DeepSurv with continuous EWC learning, the framework achieves state-of-the-art performance across all three core clinical tasks while remaining interpretable through SHAP analysis.
Dataset scale and diversity as the main limitation. The current study used public datasets totaling approximately 1,000 CT images, 500 histopathology images, and 400 genomic profiles. While this demonstrated proof-of-concept performance, clinical deployment will require validation on larger, more diverse patient populations that include rarer lung cancer subtypes, patients from different ethnic backgrounds, and data from multiple clinical institutions with different imaging equipment and protocols.
Future extension to longitudinal data. The current framework processes each patient at a single time point. A planned future direction is incorporating longitudinal data - follow-up imaging showing how tumors change over time, and serial genomic testing reflecting how cancer evolves in response to treatment. This would enable dynamic survival predictions that update as treatment progresses, providing real-time guidance for treatment modification decisions.
Toward treatment recommendation systems. The authors envision extending this framework into a full precision oncology decision support system that not only predicts subtype and survival risk but also recommends optimal treatment interventions based on those predictions. Combined with advanced explainability tools like counterfactual reasoning, such a system could help oncologists understand not just what treatment is predicted to work best, but why - closing the gap between AI prediction and clinically actionable guidance.