Lung cancer accounts for roughly 18% of all cancer-related deaths worldwide. Accurate survival prediction is essential for guiding treatment decisions, yet existing models face serious limitations when dealing with the complexity and volume of real-world patient data.
Clinician subjectivity and data heterogeneity create inconsistency. Traditional approaches rely on visual review of CT and PET imaging alongside patient records, a process that is time-intensive, prone to human error, and difficult to reproduce across different clinical settings.
Deep learning models struggle with continuous data streams. As hospitals generate new patient records daily, models need to update incrementally. Yet standard deep learning systems suffer from "catastrophic forgetting" - when they learn new data, they overwrite previously learned patterns, degrading overall accuracy.
Combining multiple data types adds another layer of difficulty. Survival prediction improves when clinical records, CT scans, PET images, and genomic (DNA) data are used together, but integrating these very different data types requires specialized architectures that most existing models lack.
HCLmNet is a Hybrid Continual Learning Multimodal Network designed to overcome the shortcomings of existing survival prediction models. It fuses imaging, genomic, and clinical data while continuously adapting to new patient information without losing what it has already learned.
The architecture combines three powerful AI components for data processing. A Swin Transformer extracts fine-grained features from CT and PET images; XLNet processes DNA mutation sequences; and a Fully Connected Network handles structured clinical data such as age, TNM stage, and smoking history.
A cross-attention fusion layer ties all data modalities together. After each data type is processed separately, this layer learns how information from one modality relates to another - for example, how a PET-detected metabolic hotspot correlates with a specific genomic mutation pattern.
The final prediction uses the Cox Proportional Hazards model to estimate 5-year survival probability. This statistical framework is a well-established method in survival analysis, and here it is integrated with deep learning to produce patient-level risk scores that can be updated as new data arrives.
Standard convolutional neural networks (CNNs) miss critical details in medical images. Models pre-trained on general image datasets like ImageNet are poorly suited for medical imaging because they focus on high-level patterns rather than subtle features such as ground-glass opacities or small multi-nodule tumors that are diagnostically important in lung cancer.
The Swin Transformer uses "shifted window attention" to detect fine local details while maintaining global context. Unlike approaches that process entire images at once, the Swin Transformer divides CT and PET volumes into small non-overlapping windows and performs attention calculations within each. Periodically shifting these windows allows cross-window dependencies to be captured too, giving the model sensitivity to tiny tumor features within a larger anatomical structure.
XLNet processes DNA mutation data using permutation-based sequence modeling. Genomic data in this dataset consists of counts of single nucleotide variants and homozygous or heterozygous mutations. XLNet learns patterns across all possible orderings of these features, capturing complex dependencies that simpler models would miss - even when mutation data is available for only a subset of patients.
All three modalities are projected to the same 64-dimensional space before fusion. This dimensional alignment ensures that clinical, imaging, and genomic embeddings can be meaningfully compared and combined by the cross-attention fusion layer, regardless of their original format or scale.
The cross-attention fusion block allows each data type to inform the representation of every other data type. For instance, a DNA mutation pattern might increase or decrease how much weight is assigned to a particular imaging feature - a relationship that cannot be captured by processing each modality independently.
Attention scores between modality pairs are computed and used to refine each representation. For every pair of data types (such as CT and DNA), the model calculates how relevant each feature in one modality is to each feature in the other, then updates both representations accordingly. This produces richer, more informative patient embeddings.
The fused embeddings pass through a Fully Connected Network before reaching the Cox Proportional Hazards layer. This additional processing step allows the model to learn non-linear relationships within the combined multimodal representation before converting it into a survival risk score.
The Cox Proportional Hazards model estimates the probability of surviving beyond 5 years for each patient. It uses the learned risk score alongside a baseline hazard function, and because it is semi-parametric, it does not assume any particular statistical distribution for survival times - making it flexible across diverse patient populations.
Catastrophic forgetting occurs when a neural network learns new data and inadvertently overwrites knowledge from earlier training. In a hospital setting where patient records are continuously added, this is a critical problem - a model that updates on new patients should not lose its ability to correctly predict outcomes for previously seen patient types.
Elastic Weight Consolidation (EWC) protects the model's most important parameters during updates. EWC uses the Fisher Information Matrix to identify which neural network weights are most critical for prior predictions. When new data is learned, updates to these critical weights are penalized, preserving the model's prior knowledge while still allowing adaptation.
Experience Replay (ER) maintains a memory buffer of 500 representative patient cases. Each time new data arrives, a subset of these stored cases is mixed into the training batch alongside the new data. This rehearsal approach ensures that the model is always trained on a blend of old and new information rather than exclusively on the most recent patients.
The loss function during incremental learning combines new-data performance with EWC regularization. The total loss penalizes deviations from previously learned parameter values in proportion to their importance scores. Hyperparameter lambda controls the strength of this penalty, allowing clinicians to tune the balance between retaining old knowledge and adapting to new patient populations.
The Instance-Level Correlation Replay (EICR) module preserves fine-grained relationships between individual patient cases across modalities. It constructs a correlation matrix capturing how similar any two patients' multimodal feature embeddings are, then ensures this relational structure remains stable as new data is added. For example, the learned association between a specific mutation pattern and a corresponding tumor intensity should not be disrupted when new patients are incorporated.
KL divergence is used to measure and minimize structural drift in instance relationships. By comparing the distribution of patient-to-patient correlations before and after a model update, the EICR module can detect when new data is causing previously learned relational patterns to shift, and corrects for this during training.
The Class-Level Correlation Replay (ECCR) module preserves distinctions between broad patient categories. It uses triplet-based contrastive learning: for each anchor patient (say, an early-stage case), a similar patient is selected as a "positive" example and a dissimilar one (late-stage) as a "negative." The model is trained to keep similar patients close together in feature space while pushing dissimilar ones apart.
Semi-hard triplet selection ensures the training signal remains meaningful. Negative examples are chosen to be moderately dissimilar - not so close that they are confusable, but not so far apart that the model learns nothing useful. This careful sample selection maintains class boundaries as new patient groups are introduced over time.
The study used a dataset of 5,053 lung cancer patients from AI Hub, a South Korean public data repository. Of these, 3,770 patients had complete multimodal data including clinical records, CT scans, PET scans, and DNA mutation data. The dataset covers survival outcomes over up to four years, with the 5-year prediction threshold used as the standard clinical benchmark.
DNA mutation data was available for only 412 of the patients, creating a significant data sparsity challenge. To address this, XLNet was pre-trained to generate latent mutation embeddings from the available cases, and these learned representations were integrated into the multimodal framework as a proxy for patients without direct genomic data.
Three sequential data subsets were used to simulate continual learning in a real hospital environment. A primary training set of 3,358 patients formed the base, and two additional cohorts of 200 and 212 patients with real genomic data were used as incremental updates. This design allowed the framework's ability to incorporate new patient data while retaining prior knowledge to be systematically evaluated.
Clinical data preprocessing involved 16 features spanning demographics, staging, and lifestyle factors. Standard techniques were applied including outlier removal, one-hot encoding of categorical variables, min-max normalization, and mean imputation for missing values. CT and PET volumes were standardized to 128x128 pixel resolution at 160 slices and augmented with flips, rotations, and Gaussian noise.
HCLmNet achieved a Concordance Index (C-index) of 0.84, substantially outperforming all baseline models. The C-index measures how accurately the model ranks patients by survival risk - a value of 1.0 would indicate perfect ranking. Traditional CoxPH and DeepSurv models scored 0.65 and 0.70 respectively, while the best multimodal model without continual learning reached 0.76.
The model reduced prediction error from 189 days to 140 days in Mean Absolute Error. This means HCLmNet's predictions of actual survival time were on average nearly 7 weeks more accurate than the base multimodal model, and roughly 37 days more accurate than the DeepSurv-based baseline, which recorded an MAE of 252 days.
Catastrophic forgetting was minimized to a forgetting score of just 0.08. This metric quantifies how much performance on older patient data degrades after the model learns new data. A score near zero indicates that the continual learning mechanisms successfully preserved prior knowledge throughout the incremental training process.
The improvements stem from the combined action of all three replay modules together with EWC. Ablation analyses confirmed that removing any single component - ER, EICR, or ECCR - reduced performance, demonstrating that the hybrid strategy is stronger than any individual approach to continual learning alone.
The TNM staging system provides the foundational clinical context for survival prediction in this framework. TNM staging - where T describes tumor size, N describes lymph node involvement, and M describes metastasis - is used globally to assess cancer severity. When combined with demographic and behavioral factors such as smoking history, it forms the basis of the clinical feature set.
Conventional models become unreliable when clinical environments evolve. As new patients enter a hospital system with different demographics, new imaging protocols, or updated genomic panels, a static model trained at a fixed point in time will gradually drift from the current patient population. Continual learning directly addresses this temporal reliability problem.
Retraining models from scratch as new data arrives is computationally expensive and clinically impractical. The hybrid approach in HCLmNet enables efficient incremental updates - preserving the investment in prior training while incorporating new patient information - making it suitable for deployment in resource-constrained hospital environments.
The 5-year survival prediction standard follows established oncology benchmarks from the American Cancer Society. This standardization allows the model's outputs to be directly compared to published survival statistics and used in treatment planning discussions, facilitating integration into existing clinical decision-making workflows.
HCLmNet represents a meaningful advance in making AI-driven survival prediction practical for real hospitals. By combining multimodal data fusion with robust continual learning, it addresses two major barriers to clinical deployment: the inability of most models to handle heterogeneous data, and the deterioration of model performance over time as patient populations change.
The framework's explainability features support responsible clinical integration. Unlike black-box models, the attention-based architecture allows clinicians to understand which data modalities or specific features most influenced a patient's predicted survival probability, supporting transparency and regulatory acceptance.
DNA data sparsity remains a limiting factor that future work could address. Only 412 of 5,053 patients had genomic mutation data, and XLNet-derived embeddings were used as a proxy for the rest. Broader genomic profiling of patient cohorts, or techniques for imputing missing genomic data, would strengthen the framework's predictive power.
Future extensions could incorporate additional modalities and longer follow-up periods. Adding pathology slide images, extended survival data beyond four years, and additional genomic markers could further improve prediction accuracy. The hybrid continual learning framework is designed to accommodate new data types without requiring complete retraining, making such extensions feasible.