This review systematically surveys deep learning approaches for multi-modal breast cancer prediction, covering 50 studies published from 2019 to 2025. The central premise is that single-modality data, whether an MRI scan, a gene expression profile, or a pathology slide, captures only a partial view of breast cancer's biological complexity. Combining data from multiple sources provides richer, more complementary information that improves the robustness and accuracy of prediction models.
The review covers the full spectrum of data types used in breast cancer research: imaging modalities (mammography, ultrasound, MRI, digital breast tomosynthesis, contrast-enhanced imaging, CT), pathological images (H&E-stained and immunohistochemical tissue slides), biological molecular data (mRNA gene expression, DNA, copy number variations, proteogenomics), and clinical data (patient records, tumor staging, treatment history). Together, these modalities span the spectrum from macroscopic tumor anatomy to molecular biology.
The review's primary analytical contribution is classifying multi-modal deep learning fusion strategies into three types: feature-level fusion (combining extracted features before the final prediction stage), decision-level fusion (combining predictions from separately trained models), and hybrid fusion (combining both strategies flexibly). The systematic analysis reveals that fusion strategy selection strongly influences model performance, with feature-level and hybrid approaches consistently outperforming decision-level fusion in capturing cross-modal complementary information.
Multi-modal breast cancer research draws on a diverse ecosystem of data types. Among imaging modalities, MRI offers the highest sensitivity (appearing in 28% of reviewed studies) but requires contrast agents and has lower specificity. Ultrasound (B-mode and contrast-enhanced variants) appears in 24% of studies, providing excellent soft tissue differentiation without radiation. Mammography (17%) remains the clinical screening standard. Digital breast tomosynthesis, contrast-enhanced spectral mammography, and CT provide additional imaging perspectives in specialized clinical contexts.
The most widely used publicly available databases include: TCGA (The Cancer Genome Atlas), a comprehensive database of 44,700 cancer cases with genome sequencing, protein, and transcriptome data; METABRIC (Molecular Taxonomy of Breast Cancer International Consortium), covering over 2,000 patients with gene expression, copy number variations, and clinical data for molecular subtype classification; TCIA (The Cancer Imaging Archive), containing over 9,000 cases with medical images and clinical information; BreakHis, comprising 9,109 breast histopathology images from 82 patients at four magnification levels; and CBIS-DDSM, an updated mammography collection with ROI segmentations and pathological diagnoses.
Clinical applications in multi-modal breast cancer research concentrate around four tasks. Breast cancer diagnosis identifies abnormal masses and assesses histological type (invasive vs. non-invasive) and molecular subtype (Luminal A, Luminal B, HER2-overexpressing, triple-negative). Neoadjuvant systemic therapy (NST) evaluation predicts whether pathological complete response will be achieved before surgery. Prognosis prediction estimates recurrence risk and survival probability post-treatment. Tumor segmentation automatically delineates tumor boundaries to support treatment planning and monitoring.
Decision-level fusion is the simplest and earliest approach: each modality is processed by an independent model, and the separate predictions are combined through averaging, weighted scoring, majority voting, or Choquet fuzzy integral methods. This approach is easy to train and preserves each modality's unique representation, but it fails to exploit complementary cross-modal information, because modalities only interact at the final prediction stage rather than during feature learning. A clinical example showed decision-level fusion of DBT and FFDM mammography achieving AUC 0.96, but losing the within-modality interaction information that richer fusion could capture.
Feature-level fusion extracts features from each modality independently, then merges them into a unified representation before the prediction stage. Early implementations combined CNN-processed image features with machine-learning-processed gene expression features (using LASSO Cox regression or random forest selection), then concatenated the feature vectors for classification. Later approaches apply deep learning to both imaging and non-imaging data, enabling unified representation learning. Attention mechanisms significantly improve feature-level fusion by allowing the model to dynamically weight the most diagnostically informative features rather than treating all features equally, with sigmoid-gated attention achieving AUC 0.95 on METABRIC.
Hybrid fusion combines feature-level and decision-level strategies, allowing multiple fusion operations within the feature extraction stage, between the feature extraction and decision stages, or both. This flexibility enables models to capture both within-modality deep features and between-modality predictive combinations. Graph-based affinity fusion combined with attention-based feature weighting achieved AUC 0.94 and F1 score 0.92 on METABRIC, among the highest performance in the reviewed studies. Adversarial training within hybrid frameworks reduces the distributional gap between modalities, as demonstrated by GAN-based approaches that convert source modality representations into target modality distributions before fusion.
Multi-modal breast cancer data presents distinct preprocessing challenges across different data types. Biological molecular data faces the curse of dimensionality: gene expression datasets may contain 30,000+ features with only hundreds of patient samples, making direct model training infeasible. Feature selection methods including filter approaches (variance thresholding, chi-squared test), wrapper methods (recursive feature elimination), embedded methods (LASSO regularization, random forests), and dimensionality reduction (PCA, mRMR) are applied to identify the most prognostically informative molecular features while eliminating redundant information.
Medical imaging data requires noise reduction, normalization, and region of interest (ROI) extraction before model training. Imaging noise from equipment limitations and operator variation can be addressed through denoising filters and anisotropic diffusion algorithms. Normalization techniques including min-max scaling and z-score standardization bring images from different scanners and protocols to a comparable range, reducing systematic bias. Automatic ROI extraction using fast region-based CNNs and region proposal networks replaces subjective manual delineation, improving efficiency and consistency while reducing inter-observer variability.
Clinical data presents a different preprocessing challenge: non-numerical categorical features such as tumor grade, molecular subtype, and treatment history must be converted into numerical representations before neural network training. One-hot encoding transforms categorical variables into binary indicator vectors, allowing models to treat each category independently without imposing arbitrary ordinal relationships. Missing value imputation using weighted nearest neighbor algorithms addresses the incomplete records common in retrospective clinical datasets. The combination of these preprocessing steps creates a standardized numerical input space that allows heterogeneous clinical, imaging, and molecular data to be processed within a unified deep learning framework.
Predicting pathological complete response (pCR) to neoadjuvant systemic therapy is one of the most clinically valuable multi-modal applications because pCR is strongly associated with long-term outcomes. Combining DCE-MRI temporal sequences with clinical data outperforms MRI or clinical data alone; one framework using 3D ResNet-50 for MRI features and fully connected layers for clinical data achieved significantly improved pCR prediction. Multi-modal approaches integrating ultrasound radiomics, proteomic data, transcriptomic data, and clinical features have consistently outperformed single-modality baselines, with the concordance of multi-modal predictions providing oncologists with more reliable guidance for treatment planning decisions.
For prognosis prediction (survival and recurrence risk), multi-modal approaches that integrate gene expression, DNA methylation, microRNA, and copy number variation achieve C-index improvements of approximately 0.025 over the best single-modality models. Clinical multi-modal frameworks combining pathological image features from ResNet with clinical data selected via chi-squared test achieved strong breast cancer prognosis prediction performance. Models that also incorporate immune cell data and spatial tumor microenvironment features, such as tumor-infiltrating lymphocyte density, add independent predictive value beyond classical staging systems. Survival analysis consistently shows that multi-modal risk scores stratify patients into significantly different outcome groups.
Tumor segmentation in 3D medical images represents a particular challenge because tumors occupy a small fraction of the total volume, and boundary information is often blurred. Multi-modal CT fusion combining monoenergetic images, iodine-no-water maps, and effective atomic number maps achieved specificity of 0.94 and sensitivity of 0.84 for lesion segmentation, outperforming single-modality approaches. Weighted multi-modal U-Net architectures for automatic lesion segmentation in ultrasound enable downstream classification without manual delineation, eliminating the subjectivity and time burden that has historically limited the scalability of AI-assisted pathology workflows.
The most significant structural challenge in multi-modal breast cancer AI is the scarcity of high-quality public datasets that include matched data across multiple modalities. Most publicly available datasets provide rich information in only one or two modalities; very few include simultaneously available imaging, pathology, genomics, and clinical data for the same patients. This scarcity forces researchers to use smaller, institution-specific cohorts that limit generalizability and make cross-study performance comparisons unreliable.
At the architectural level, current multi-modal fusion methods face limitations in adaptively managing the heterogeneity between modalities. Image and non-image data have fundamentally different dimensionalities, distributions, and information densities. Homogeneous fusion networks that apply the same architecture to all modalities struggle to accommodate these differences; heterogeneous networks that customize architectures per modality require careful design and are harder to train. Attention mechanisms, adversarial alignment, and graph-based cross-modal integration are actively being developed to address these challenges, but no universal fusion strategy has emerged as optimal across all breast cancer prediction tasks.
Future research priorities include developing open-access multi-modal breast cancer databases with matched imaging, molecular, and clinical data; creating adaptive fusion strategies that automatically select the most appropriate fusion method based on the characteristics of the specific modalities and tasks available; and building more robust network architectures that maintain performance when certain modalities are unavailable at inference time. The integration of transformer-based cross-modal attention mechanisms and large language models for clinical text processing represents the most promising near-term pathway toward AI systems that can synthesize the full diversity of information available in modern breast cancer care.