Around 30% of breast cancer cases will eventually recur and spread to distant organs, even after appropriate initial treatment. For patients with metastatic disease, the 5-year survival rate falls below 23%, making accurate recurrence risk prediction a critical priority for guiding treatment decisions.
Traditional approaches -- such as standard biomarkers, clinical imaging, and molecular tests -- have real limitations: low sensitivity, high cost, limited availability, and an inability to fully capture the complexity of individual tumors. A promising alternative is multi-modal data integration, which combines different types of patient information to build a richer picture of disease biology.
This study introduces the DeepClinMed-PGM model, a deep learning framework that integrates three data streams -- digital pathology images, gene expression profiles, and clinical characteristics -- to predict disease-free survival (DFS), the time a patient lives without cancer returning after treatment.
The study enrolled 1,020 patients with non-metastatic breast cancer from two sources: The Cancer Genome Atlas (TCGA), a major US public database, and Sun Yat-sen Memorial Hospital (SYSMH) in Guangzhou, China. Patients were divided into a training cohort (741 patients), an internal validation cohort (184 patients), and an external testing cohort (95 patients) to rigorously evaluate performance.
Pathology data consisted of whole slide images (WSIs) -- digitized, high-resolution scans of tissue specimens stained with hematoxylin and eosin. These images were processed by dividing them into 256x256 pixel patches and extracting features using a ResNet50 deep learning model pre-trained on the ImageNet image database. This transfer learning approach allows the model to recognize complex visual patterns without needing labeled annotations on every patch.
Molecular data consisted of RNA sequencing profiles spanning over 30,000 genes. From this vast dataset, 219 genes were selected based on their statistically significant association with survival outcomes. Immune cell composition was also quantified using the XCell computational tool, identifying 64 immune and stromal cell types from gene expression signatures.
The model operates in two stages: feature extraction and survival prediction. In the first stage, a modified ResNet50 architecture processes each image patch from a whole slide, generating 1,024-dimensional feature vectors that encode visual tissue characteristics. Up to four graphics processing units (GPUs) were used in parallel to handle the computational demands of processing thousands of patches per patient.
A self-attention module then assigns importance weights to each patch, allowing the model to focus on the most diagnostically relevant regions of the slide rather than treating all tissue areas equally. This attention mechanism produces a single, summarized feature vector representing each patient's entire pathology slide, highlighting areas most likely to carry prognostic information.
In the second stage, the slide-level features are combined with the 219 prognostic genes, immune cell abundance data, and clinicopathological characteristics (including PAM50 molecular subtype, TNM staging, and age) through a deep survival model architecture. The model outputs three values -- two survival probabilities and an estimated survival time -- whose product yields a final multimodal risk score for each patient.
Adding multi-modal data dramatically improved prediction accuracy compared to using pathology images alone. In the training cohort, the area under the ROC curve (AUC) -- a measure of how well a model distinguishes high- and low-risk patients -- reached 0.979, 0.957, and 0.871 for 1-, 3-, and 5-year DFS predictions, respectively.
Critically, the model performed well even in the external testing cohort from a different hospital and country, achieving AUC values of 0.851, 0.878, and 0.938 for 1-, 2-, and 3-year predictions. The C-index, another measure of how well the model ranks patients by risk, reached 0.925, 0.823, and 0.864 across the three cohorts, consistently outperforming models using only pathology images, gene signatures, or clinical data alone.
Patients classified as high-risk by the model had dramatically worse DFS than those classified as low-risk, with hazard ratios of 0.027, 0.117, and 0.061 across the three cohorts (all P less than 0.0001). Decision curve analysis further confirmed the model's clinical utility, showing a net benefit over all competing approaches across a wide range of clinical decision thresholds.
Analysis of the 219 prognostic genes revealed enrichment in pathways related to immune regulation, including T-cell activation, monocyte differentiation, and histone modification. The two most frequently mutated genes in the high-risk group were KMT2C (11%) and MAP3K1 (10%), both linked to cancer progression and altered in multiple cancer types.
Immune cell infiltration differed significantly between high- and low-risk patients. The high-risk group showed elevated levels of M2 macrophages, which are associated with immune suppression and poorer outcomes, while the low-risk group had higher expression of immune checkpoint genes and immune factor-related genes, suggesting a more active anti-tumor immune response.
Attention heatmaps generated using the GradCAM algorithm provided a visual interpretation of the model's predictions. In high-risk patients, the model identified concentrated hotspots both within and distant from the primary tumor region -- potentially capturing signs of microscopic spread -- while low-risk patients showed attention concentrated primarily within the main tumor mass.
Importantly, the DeepClinMed-PGM risk score remained a statistically independent predictor of DFS even after controlling for established clinical variables such as age, TNM stage, T stage, and N stage. While those clinical variables were significant in univariate analysis, only the model's risk score retained strong independent significance in multivariate analysis (P less than 0.001).
The model maintained high predictive accuracy across all major molecular subtypes of breast cancer, including Luminal A, Luminal B, HER2-enriched, basal-like, and triple-negative subtypes. This robustness across biologically diverse tumor types suggests the model captures fundamental prognostic features not fully captured by subtype classification alone.
From a clinical workflow perspective, the model integrates data routinely collected before surgery -- pathology slides from biopsies, RNA sequencing, and clinical records -- making it potentially feasible to run as a preoperative decision-support tool. This timing would allow risk stratification to inform surgical planning, neoadjuvant treatment decisions, and post-treatment surveillance intensity.
The DeepClinMed-PGM framework represents a meaningful advance in precision oncology -- the use of a patient's individual biological data to guide personalized treatment. By combining the visual information in tissue slides with the molecular information in gene expression profiles and the contextual information in clinical records, the model achieves a more complete picture of prognosis than any single data type can provide.
The authors acknowledge key limitations. The external validation cohort was relatively small (95 patients), and real-world deployment would require prospective validation across larger, more diverse populations. The computational pipeline for RNA sequencing and whole slide image processing is also technically demanding, which may limit adoption in lower-resource settings.
Looking ahead, expanding the model to integrate additional data modalities -- such as radiology imaging, proteomics, or treatment response data -- could further improve accuracy. The ultimate goal is a clinically deployed tool that helps oncologists personalize treatment intensity, select appropriate systemic therapies, and identify patients who may benefit most from clinical trial enrollment.