Breast cancer is the most common cancer in women globally, accounting for over 11.6% of all cancer cases. Early detection is critical: the five-year survival rate for early-stage breast cancer is approximately 90%, compared to only 15% for metastatic disease. Non-invasive imaging is the cornerstone of screening, with mammography (MGI) and ultrasound (USI) being the two most widely used modalities for automated detection systems.
Each modality has distinct strengths and weaknesses. Mammography provides detailed structural images and is particularly effective at identifying microcalcifications -- tiny calcium deposits often associated with early-stage malignancies -- but it involves low-dose radiation and can struggle to distinguish benign from malignant masses in dense breasts. Ultrasound uses high-frequency sound waves, involves no radiation, and excels at distinguishing solid from fluid-filled lesions and providing soft tissue detail, but lacks the structural resolution of mammography.
A multimodal approach combining both modalities offers the potential to overcome the individual limitations of each. Mammography's precision in structural detail complements ultrasound's ability to characterize tissue composition, creating a more complete diagnostic picture. Most existing AI systems rely on a single modality, missing this complementary information. This study proposes a novel lightweight deep learning architecture that leverages both mammogram and ultrasound data simultaneously for improved breast cancer classification.
The study assembles a large multimodal dataset from seven distinct public sources. Three datasets contribute mammogram images (MGI): the DDSM (Digital Database for Screening Mammography, 10,239 images), MIAS (322 images), and INbreast (410 images). To maintain class balance, 5,000 benign and 5,000 malignant images were selected from the combined MGI sources, totaling 10,000 mammogram images.
Four datasets contribute ultrasound images (USI): BrEaST (256 images), BUSI (780 images), Thammasat (263 images), and HMSS (2,006 images). A balanced selection of 1,500 benign and 1,500 malignant ultrasound images was taken from these sources, totaling 3,000 USI images. The combined initial dataset thus contains 13,000 images across both modalities, with equal representation of benign and malignant pathology types.
Preprocessing standardized all images to 256x256 pixels and applied Box-Cox transformation for normalization -- a statistical technique that handles non-normally distributed pixel intensity data better than standard normalization methods. Because medical images from different acquisition devices and clinical settings can have widely varying intensity distributions, Box-Cox normalization reduces these systematic biases before model training, helping ensure the network learns diagnostic features rather than scanner-specific artifacts.
A key challenge in the study is the significant size imbalance between the two modalities: 10,000 mammograms versus only 3,000 ultrasound images. Training a neural network on such unbalanced data risks the model learning more robust features from the larger mammogram set while underperforming on ultrasound inputs. To address this, the study applies StyleGAN3, an advanced generative adversarial network, to synthesize 10,000 additional high-quality ultrasound images.
StyleGAN3 is an improvement over earlier GAN architectures, introducing an alias-free design that prevents the checkerboard artifacts common in previous GAN outputs. It uses a mapping network that transforms random noise vectors into a structured intermediate latent space (called W), which then controls the synthesis network through adaptive instance normalization (AdaIN) -- a technique that separately adjusts the scale and shift of each layer's feature maps to produce diverse, high-fidelity outputs. The model is also trained using a progressive growing technique, starting at low resolution and gradually adding layers for higher resolution output.
Separate StyleGAN3 models were trained for each lesion type (benign and malignant) to ensure synthetic images accurately captured the distinguishing characteristics of each category. Training ran until the model had processed one million real ultrasound images, ensuring thorough learning of subtle textural variations. Quality was evaluated using the Frechet Inception Distance (FID) metric: malignant synthetic images achieved an FID of 22.9 and benign images achieved 35.4, both indicating high similarity to real ultrasound data. After augmentation, the ultrasound training set grew from 2,400 to 12,400 images, creating a total training dataset of 20,400 images.
The proposed LightweightUNet model is a compact encoder-decoder architecture containing only nine layers total: four encoder layers, one bottleneck layer, and four decoder layers. The defining feature of the architecture is its use of Depthwise Separable Convolutions (DSConv) throughout, which replace standard convolutions with a two-step process that dramatically reduces computational cost.
Standard convolution applies a single filter across all input channels simultaneously, which is computationally expensive. DSConv first performs depthwise convolution -- applying a separate filter to each individual channel in parallel, capturing spatial features within each channel independently. It then applies pointwise convolution (a 1x1 filter) to combine information across channels. This two-step factorization achieves the same representational power as standard convolution at a fraction of the computational cost, making the model suitable for deployment on portable devices or in resource-limited clinical environments.
The architecture follows the U-Net pattern with skip connections linking corresponding encoder and decoder layers. During encoding, the network progressively downsamples and doubles the number of feature channels, building a hierarchical representation from fine local details to broad global patterns. During decoding, it progressively upsamples while receiving detailed spatial information from the encoder via skip connections, allowing it to reconstruct precise spatial localization of findings. A global average pooling layer followed by a fully connected layer produces the final binary classification (benign versus malignant).
Two experiments were conducted under identical conditions using 5-fold cross-validation. The first experiment used the real dataset only (13,000 images, with 10,400 for training). Across all five folds, the LightweightUNet achieved an average accuracy of 86.87%, precision of 87.16%, recall of 86.87%, and F1-score of 86.84%. Performance was consistent across folds, with fold 4 achieving the highest individual accuracy at 89.73%, suggesting the model learns reliably across different data subsets.
The second experiment used the combined real plus GAN-augmented dataset (20,400 training images). The model achieved an average accuracy of 96.35%, precision of 96.36%, recall of 96.35%, and F1-score of 96.35%. These results represent improvements of approximately 9.48% in accuracy, 9.20% in precision, 9.48% in recall, and 9.51% in F1-score compared to the real-data-only experiment -- a substantial and consistent improvement across all four metrics.
The magnitude of improvement is notable given that the augmentation was applied only to the ultrasound portion of the dataset, not to the mammograms. This suggests that the model's performance was previously limited by the small and unrepresentative ultrasound training set, and that expanding USI data with high-quality synthetic examples allowed the model to learn substantially better ultrasound features. The balanced improvement across all metrics, rather than just one, indicates that GAN augmentation helped the model more reliably distinguish both benign and malignant cases.
A distinctive feature of this study is its emphasis on deployability in resource-constrained settings. Breast cancer screening rates are significantly lower in low- and middle-income countries, partly due to the cost and infrastructure requirements of high-end imaging equipment and powerful computing hardware for AI analysis. A lightweight model that can run on portable devices or low-cost hardware could dramatically expand access to AI-assisted screening beyond well-equipped urban hospitals.
The LightweightUNet's nine-layer architecture with depthwise separable convolutions requires significantly less GPU memory and processing time than competing deep learning approaches such as ResNet-50, VGG-16, or full U-Net architectures. The study's hardware requirements (16 GB RAM, 12 GB GPU) are already substantially lower than many competing methods, and the compact architecture is designed with mobile device deployment in mind.
The multimodal approach also has practical clinical advantages beyond improved accuracy. In settings where only one imaging modality is available, single-modality results can still be used. In settings with both mammography and ultrasound, combining both provides superior decision support. This flexibility makes the approach adaptable to different clinical workflows and resource availability, potentially allowing gradual adoption as imaging infrastructure improves.
The study makes several interconnected contributions. First, it introduces the LightweightUNet architecture, which achieves strong classification performance with minimal computational overhead. Second, it demonstrates a robust multimodal training approach using 13,000 images from seven public datasets, ensuring the model is exposed to diverse imaging characteristics from multiple institutions and scanner types. Third, it provides a systematic comparison of performance with and without GAN augmentation, quantifying the benefit of synthetic data for a specific modality imbalance problem.
The 9.48% accuracy improvement from GAN augmentation illustrates a general principle: when one data modality is significantly underrepresented compared to another, synthetic data generation can efficiently close the gap without requiring additional clinical data collection. The use of StyleGAN3 specifically, with its alias-free design and progressive growing, produced higher-quality synthetic images (lower FID scores) than older GAN architectures would have achieved, contributing to the magnitude of improvement.
Future research directions include extending the model to multi-class classification across cancer stages or subtypes, evaluating performance on external validation datasets from different clinical sites, and implementing formal model interpretability techniques (such as Grad-CAM attention maps) to help clinicians understand which image regions drive the AI's classification decisions. Prospective clinical validation studies comparing the model's performance against radiologist assessment would be the essential next step before any clinical deployment.