Breast cancer is the most prevalent malignant cancer among women worldwide. A pivotal question in treatment planning is whether cancer cells have spread to the axillary lymph nodes (ALN) -- the lymph nodes in the armpit that represent the earliest route through which breast cancer disseminates to distant organs.
The current standard for determining lymph node status is sentinel lymph node biopsy (SLNB), an invasive surgical procedure that carries risks of lymphedema, seroma, and shoulder dysfunction. Preoperative prediction of lymph node status could allow surgeons to avoid unnecessary SLNB in low-risk patients while planning more aggressive treatment for those predicted to have spread.
Prior prediction tools, including the widely-used Memorial Sloan-Kettering Cancer Center (MSKCC) nomogram, achieved AUCs around 0.75 using clinical variables alone. These approaches predicted only whether metastasis was present or absent, without distinguishing the degree of spread -- a clinically important gap that this study set out to fill.
This study developed a multi-modal multi-instance (MMMI) deep learning model that jointly analyzes two types of information obtained from preoperative core needle biopsy specimens: digitized whole-slide histopathology images (WSIs) and structured clinicopathological data (age, tumor size, hormone receptor status, TILs, etc.).
The primary dataset included 3,701 female breast cancer patients from the Fourth Hospital of Hebei Medical University, collected between January 2015 and December 2018. An external validation cohort of 190 patients was drawn from four additional medical centers in Hebei Province, enabling a true multicenter assessment of generalizability.
A key innovation of this study was its focus on four-class lymph node classification -- distinguishing node-negative (N0), isolated tumor cells (ITCs), micrometastasis, and macrometastasis -- rather than the simpler binary present/absent prediction used in most prior work. This granularity matters because clinical management differs significantly across these categories.
Whole-slide images (WSIs) of core needle biopsy specimens are extremely large files -- typically 130,000 by 50,000 pixels -- that cannot be processed directly by standard neural networks. The study used multi-instance learning (MIL), in which each WSI is divided into smaller 512x512-pixel patches (instances) that are analyzed collectively to generate a slide-level prediction.
The model extracted patch-level features using EfficientNet pre-trained on ImageNet, then applied attention mechanisms at both the instance level and feature level to learn which patches contributed most to the final lymph node status prediction. Patches were processed at three magnification scales (5x, 10x, and 20x) to capture both cellular details and tissue-level architecture simultaneously.
Cancer tissue regions within each WSI were first annotated and extracted, then tiled into patches for feature extraction. The use of multiple magnification scales was designed to provide complementary information: low magnification captures tumor architecture and microenvironment, while high magnification reveals nuclear and cellular features most relevant to aggressive behavior.
The core innovation of MMMI is its Multi-modal Multi-instance Fusion (M3IF) module, which creates a bidirectional information exchange between the image and clinical data branches rather than simply concatenating them at the end. Clinicopathological data is processed using TabNet -- an attentive interpretable tabular learning network that learns which clinical features are most informative at each decision step.
The fusion works in two directions: clinical data features are used to guide the attention process in the image branch, helping the model focus on patches whose characteristics align with the patient's clinical profile; and image-derived features simultaneously recalibrate the representation of clinical data. This bidirectional cross-modal recalibration is designed to capture complementary information that neither data source could provide alone.
After recalibration, the refined image features from all three magnification scales and the refined tabular features are concatenated and passed through a multi-layer fully connected network to produce the final four-class prediction probability. The entire model was trained end-to-end, with label smoothing and weighted sampling applied to handle the imbalanced distribution of lymph node status categories.
For the fundamental binary task of predicting whether lymph node metastasis was present or absent, the MMMI model achieved an AUC of 0.809 on the internal test set. This outperformed the clinical-data-only model (AUC 0.770) and the WSI-only model (AUC 0.709), confirming that multimodal fusion provides meaningful improvement over either single-source approach.
MMMI also achieved the best accuracy (0.751), balancing sensitivity (0.768) and specificity (0.734) more evenly than either single-modality model. The WSI-only model showed high specificity (0.757) but poor sensitivity (0.593), while the clinical-only model had the reverse pattern, suggesting that the two modalities capture complementary aspects of metastatic risk.
In the external multicenter validation set, MMMI achieved a binary AUC of 0.6258 -- a notable decline from the internal test set performance. The authors attribute this gap primarily to differences in slide preparation, H&E staining protocols, and interpretation conventions across the five institutions involved, highlighting the challenge of cross-site generalization in histopathology AI.
The four-class classification -- distinguishing node-negative, isolated tumor cells (ITCs), micrometastasis, and macrometastasis -- showed MMMI outperforming both single-modality models across all categories. For macrometastasis, MMMI achieved AUC 0.758 vs. 0.748 (clinical only) and 0.691 (WSI only). For micrometastasis, MMMI reached 0.691, substantially higher than clinical-only (0.636) and WSI-only (0.617).
ITC prediction was the most difficult category, as expected given the very small number of cases (118 out of 3,701 patients, or just 3%) and the subtle microscopic changes associated with isolated tumor cells. MMMI nonetheless achieved AUC 0.634 for ITCs -- higher than clinical-only (0.619) and substantially higher than WSI-only (0.531), which performed poorly on this rare category.
In the four-center external validation, MMMI achieved AUCs of 0.725 (no metastasis), 0.757 (ITCs), 0.525 (micrometastasis), and 0.708 (macrometastasis). The low micrometastasis AUC was attributed primarily to the very small number of external validation cases (only 8 micrometastasis patients), making this estimate unreliable rather than indicative of true model weakness.
When model performance was stratified by molecular subtype, MMMI showed notably different accuracy across Luminal, HER2-overexpressing, and triple-negative breast cancer (TNBC) subtypes. For the no-metastasis class, TNBC had the highest AUC at 0.895, while Luminal and HER2-positive tumors achieved 0.784 and 0.885 respectively.
For micrometastasis prediction, TNBC again led with AUC 0.968 -- a striking result given that TNBC is generally the most aggressive and difficult to treat subtype. The authors suggest this is because TNBC tumors have more distinctive imaging and pathological features (high mitosis rates, high TILs, high nuclear grade) that make their metastatic behavior more predictable from biopsy tissue.
The finding that AI models can predict lymph node status with particularly high accuracy in TNBC is clinically relevant because TNBC patients typically face the most difficult treatment decisions -- they do not respond to hormone therapy or HER2-targeted drugs, making surgical and systemic treatment optimization especially important. A model that works best in precisely this high-stakes group is particularly valuable.
To understand which biological features drove predictions, the model's learned attention weights were analyzed as a form of interpretability analysis. The most important clinicopathological feature was mitosis count -- the rate at which cancer cells are actively dividing -- followed by glandular formation (reflecting tumor differentiation), estrogen receptor (ER) status, stromal changes, vascular invasion, Ki67 proliferation index, and tumor-infiltrating lymphocytes (TILs).
These findings align with well-established biological knowledge: tumors with high mitosis rates are more aggressive and more likely to spread; absence of glandular structures (poor differentiation) reflects more primitive cancer cells; and vascular invasion directly indicates that cancer cells have entered blood or lymphatic vessels -- a prerequisite for lymph node colonization.
The inclusion of TILs in the top predictors is particularly interesting because TILs represent the immune system's response to the tumor. Paradoxically, TNBC tumors often have high TILs, yet this is also the subtype where the model performed best, suggesting the model captured the complex interplay between immune infiltration and metastatic biology rather than relying on simple linear relationships.
Most prior deep learning studies for lymph node prediction from histopathology focused on binary classification and single-center datasets. This study is distinguished by its four-class classification scheme aligned with the AJCC staging system (N0, ITC, N1mi, N1), its very large primary cohort of 3,701 patients, and its multicenter validation across five institutions.
The cross-modal recalibration architecture is a meaningful methodological advance over simple feature concatenation. By letting clinical data actively guide which image patches the model attends to, the MMMI framework makes the image analysis more targeted and clinically grounded -- rather than having the model discover the same patterns from images alone that are already captured in the clinical record.
The performance decline in external validation reflects a real-world challenge in digital pathology AI: H&E staining protocols, scanner characteristics, and pathologist annotation conventions vary substantially between hospitals, creating domain shift that degrades model performance. The authors identify data standardization across centers as a critical prerequisite for further improving external generalization.
This study demonstrated that combining clinicopathological data with whole-slide histopathology images from core needle biopsies in a multimodal deep learning framework can predict breast cancer lymph node status more accurately than either data source alone. The MMMI model achieved AUC 0.809 for binary and competitive performance across the four-class prediction in both internal and external cohorts.
Key limitations include the relatively small number of ITC and micrometastasis cases compared to node-negative and macrometastasis cases, which restricted the model's ability to learn the subtle features distinguishing these intermediate categories. The ITC and TNBC subtype analyses were particularly limited by case counts, and the estimated confidence intervals for some subgroups were too wide to draw firm conclusions.
Future plans include expanding the dataset by incorporating additional center data and genetic test results, optimizing the slide scanning and staining protocols to reduce inter-center variability, and exploring the inclusion of immunohistochemical stains beyond H&E. The study's code is publicly available via the Tencent AI Lab GitHub repository, supporting further development and replication by the research community.