Breast cancer is the most common malignancy among women worldwide. Mammography remains the established standard for population-based screening due to its high spatial resolution and ability to detect early indicators like microcalcifications, but its sensitivity drops substantially in women with dense breast tissue. Ultrasound complements mammography by differentiating cystic from solid lesions without radiation, though its accuracy is highly operator-dependent. MRI offers superior soft-tissue contrast and sensitivity for invasive cancers but is limited by cost, access, and lengthy examination times.
Convolutional neural networks (CNNs) have been widely adopted for breast imaging AI, with applications in lesion segmentation, classification, detection, and risk stratification. Architectures such as U-Net for segmentation and deep residual networks for classification have shown strong performance. However, CNNs rely on large annotated datasets -- which are expensive to produce and often restricted by privacy regulations -- and are limited to local receptive fields, making them less effective at capturing long-range spatial relationships across entire images.
Self-supervised learning (SSL) has emerged as a label-efficient alternative that reduces dependence on manual annotation. SSL trains models on unlabeled data using surrogate or 'pretext' tasks -- such as predicting masked image regions, rotation angle, or contrastive alignment of augmented views -- to learn rich feature representations. These pretrained models can then be fine-tuned on smaller labeled datasets, achieving competitive performance while dramatically reducing annotation requirements.
Vision Transformers (ViTs) address a core limitation of CNNs by using multi-head self-attention to model relationships between all image regions simultaneously, capturing global contextual information from the earliest processing layers. Hybrid architectures that combine CNN backbone feature extraction with transformer global attention have emerged as a particularly powerful approach, combining the local texture sensitivity of convolutional layers with the long-range contextual reasoning of self-attention.
This narrative review systematically searched PubMed, Scopus, IEEE Xplore, and Web of Science for studies published between January 2015 and June 2025. From 761 identified records, 85 peer-reviewed studies met the inclusion criteria of applying SSL or transformer-based architectures to breast cancer imaging tasks including classification, detection, or segmentation across mammography, ultrasound, and MRI modalities.
SSL methodologies in medical imaging are grouped into three categories. Contrastive methods (SimCLR, MoCo) maximize agreement between differently augmented views of the same image while minimizing similarity to unrelated samples. Non-contrastive approaches (BYOL, SimSiam) use dual-network architectures to align representations without requiring negative pairs. Generative strategies -- including denoising autoencoders and generative adversarial networks -- model the underlying data distribution through reconstruction or synthesis, improving robustness to noise and class imbalance.
Vision Transformers process images by partitioning them into fixed-size patches, embedding them as token sequences, and applying stacked transformer encoder layers with multi-head self-attention. This design enables global context modeling from early stages but lacks the inductive biases of CNNs (locality, hierarchical feature extraction), making ViTs typically data-hungry. Hierarchical variants like the Swin Transformer address this by applying local window-based self-attention with a shifted-window mechanism for cross-window communication, reducing computational complexity from quadratic to linear.
Hybrid CNN-Transformer architectures combine the complementary strengths of both paradigms within a unified framework. Sequential designs use CNNs to extract multi-scale feature maps that are then tokenized for transformer processing. Dual-stream architectures run CNN and transformer branches in parallel and merge outputs through learned fusion. A third approach embeds lightweight attention modules within CNN layers. All three variants consistently outperform pure CNN or pure transformer architectures in breast imaging benchmarks.
For breast lesion segmentation, SSL has demonstrated consistent gains across all imaging modalities. In mammography, a BYOL-based transformer achieved 16.7% improvement in validation accuracy over supervised baselines. In ultrasound, Closely Related SSL (CR-SSL) achieved Dice scores up to 82.1% with 10-20% improvement over supervised CNNs. In MRI, a dual-module SSL approach combining global content perception with peritumoral context restoration achieved 87.8% Dice and outperformed nine SSL and semi-supervised baselines on 229 DCE-MRI cases. Weakly supervised SSL methods using only scribble annotations achieved within 10-15% of fully supervised segmentation accuracy.
For detection tasks, SSL proxy tasks such as rotation prediction and image reconstruction generally outperform contrastive variants in breast imaging, producing orientation-invariant and anatomically meaningful features. A multi-task SSL U-Net for automated breast ultrasound achieved 79.6% recall with only 10% of voxel-level labels. In mammography, a modified YOLOv8 with SSL pretraining achieved 99.3% precision and 85.0% recall for calcification detection. In MRI, self-supervised rotation learning achieved 95.8% AUC and 92.5% accuracy -- outperforming ImageNet transfer learning -- on binary cancer vs. healthy tissue classification.
Semi-supervised hybrid approaches show that combining SSL pretraining with limited supervision can approach fully supervised performance. CT-Match, a dual-branch CNN-Transformer framework, achieved Dice scores of 78.9% and 84.9% on two ultrasound datasets using far fewer labeled examples than standard supervised training. These results collectively demonstrate that SSL reduces annotation burden while maintaining clinically acceptable performance across diverse breast imaging scenarios.
Despite these advances, significant challenges persist. Most SSL studies rely on retrospective single-center datasets, limiting generalizability across scanner hardware, acquisition protocols, and patient demographics. External validation was inconsistently reported across studies. SSL proxy tasks may also struggle to capture rare or subtle pathologies, suggesting that combining multiple SSL pretext tasks or integrating domain-specific priors could enhance sensitivity for uncommon lesion presentations.
For breast lesion segmentation, transformer-based models consistently outperform CNN-only counterparts. In mammography, the Swin Transformer-based Swin-SFTNet improved Dice by 3.1-3.8% on CBIS-DDSM and INbreast. The TrEnD encoder-decoder architecture achieved Dice scores exceeding 92% with IoU above 85% on both datasets. In MRI, the TraBS model achieved 91.6% Dice internally and 86.4% externally for fibroglandular tissue segmentation, demonstrating cross-institutional robustness. In ultrasound, the HCTNet hybrid CNN-transformer achieved up to 97.23% Dice on a high-quality dataset, though performance on standard BUSI images was lower (82%), reflecting the impact of image quality variability.
For breast lesion detection, cross-view and cross-modal transformer designs show particular advantage. TransReg -- combining a Swin Transformer backbone with a cross-transformer registration module and Faster R-CNN detection head -- achieved 83.3% recall on DDSM and 79.7% on VinDr-Mammo by learning to compare bilateral breast images, mirroring how radiologists assess asymmetry. The Multi-Modal Transformer (MMT) trained on 1.3 million screening exams achieved 94.3% AUC for cancer detection and 82.6% for 5-year risk prediction, demonstrating that transformer performance scales favorably with dataset size.
For breast lesion classification, hybrid CNN-Transformer models achieve standout results on mammography. A residual CNN plus Transformer encoder combination achieved 100% binary classification accuracy and 95.8% multiclass accuracy on CBIS-DDSM. An ensemble CNN with ViT encoder reached 98.58% binary accuracy on INbreast. In ultrasound, multi-stage pretraining -- first on ImageNet, then on cancer-cell histology, then on ultrasound -- yielded AUCs of 100% on the Mendeley dataset, 96.8% on BUSI, and 93.7% on mixed data. The multimodal MMST-V integrating DCE-MRI with clinical and geometrical data achieved 92.8% AUC, substantially exceeding imaging-only (86.3%) or clinical-data-only (90%) baselines.
Across detection, segmentation, and classification tasks, transformer models consistently yield 3-6% gains in AUC, recall, and F1 over leading CNN architectures. They also offer practical advantages: models like TransReg can compare bilateral mammographic views without manual alignment, and multi-modal transformers can incorporate heterogeneous data types (images, EHR, radiology reports) within a single unified architecture, enabling richer contextual representations than single-modality CNN approaches.
The most powerful recent approaches combine SSL pretraining with transformer-based architectures, addressing complementary limitations of each approach independently. SSL provides label-efficient feature learning from large unlabeled datasets. Transformers provide global contextual modeling that enhances diagnostic accuracy. Together, they enable models that learn richer representations without requiring the large annotated datasets that both fully supervised CNNs and supervised transformers demand.
Representative hybrid SSL-Transformer approaches span all imaging modalities. In ultrasound, PolyBreastVit and C-TUNet achieve accuracies exceeding 95% across benign, malignant, and normal classes. The DSMT-Net (Dual Self-Supervised Multi-Operator Transformation) framework achieves 5-7% accuracy gains over supervised CNN baselines by jointly leveraging labeled and unlabeled data through dual pretext task learning. BCT-Net combines CNN feature extraction with transformer-based semantic contrastive alignment, achieving 88.7% Dice in simultaneous segmentation and classification.
For high-resolution mammography, pyramid transformer architectures with masked self-attention and hierarchical attention mechanisms achieve near-perfect benign vs. malignant classification on established benchmarks. The PatchCascade-ViT and CascadePLS-ViT approaches using SSL pretraining achieve 85% sensitivity and F1 on BI-RADS classification across 4,368 mammograms, outperforming CNN baselines in both breast density and cancer risk categorization. Spatially adaptive tokenization -- directing more computational resources to diagnostically significant regions -- further improves efficiency without accuracy tradeoffs.
Multi-task learning within hybrid frameworks enables simultaneous optimization of complementary tasks such as segmentation and classification. YOLOv5 enhanced with a LOGO Transformer achieves both detection (95.7% true positive rate) and segmentation (F1 74.5%) on the same model pass. The VMamba architecture applied to breast ultrasound achieves 96.96% AUC for classification and 85.59% mean IoU for segmentation, demonstrating that multi-task transformers can approach single-task specialist performance across both objectives simultaneously.
Despite strong benchmark performance, significant gaps remain between current results and clinical readiness. Most studies use retrospective single-center datasets that do not capture the heterogeneity of clinical practice: variations in scanner hardware, acquisition protocols, imaging artifacts, and patient demographics all cause performance degradation when models are deployed outside their training environment. Prospective clinical trials with blinded reader studies and integration into real radiology workflows (PACS/RIS infrastructure) are essential prerequisites for regulatory approval and deployment.
Model interpretability represents a critical barrier to clinician trust. While attention maps from Vision Transformers can highlight image regions influencing predictions, few studies have systematically evaluated their transparency in clinical workflows. Comparable visualization techniques like Grad-CAM and SHAP are available for CNNs but remain underutilized. Providing attention heatmaps, saliency overlays, and formal explainability frameworks aligned with radiologists' decision criteria is essential for models to function as trusted decision support tools rather than opaque black boxes.
Computational demands of large-scale transformer models (ViT-L, Swin-B) present practical deployment challenges in resource-limited or time-sensitive settings. Future work should prioritize lightweight transformer variants, model compression techniques, and optimized attention mechanisms that enable real-time inference without accuracy sacrifice. Federated SSL frameworks -- which train across geographically distributed datasets while preserving patient privacy -- represent a promising approach to overcoming data-sharing restrictions that currently limit dataset scale and diversity.
The most promising future direction is multimodal SSL-Transformer integration that jointly leverages mammography, ultrasound, and MRI along with clinical data, genetic markers, and radiology reports within unified foundation models. The creation of standardized benchmarks and publicly available multi-institutional datasets specifically designed for SSL and weakly supervised learning will be critical for enabling reproducible performance comparisons. SSL-driven AI has the potential to make advanced diagnostic capabilities scalable and cost-effective across diverse global healthcare settings, but only if matched by equal investment in clinical validation, interpretability, and workflow integration.