Prostate cancer diagnosis and grading depend heavily on analyzing whole-slide pathology images (WSIs), which are massive digital scans of tissue samples. The standard method for grading disease severity is the Gleason grading system, which scores cancer based on the shape and organization of prostate glands.
Within the Gleason system, pathologists distinguish low-risk glands (Gleason grade 3, which are well-formed) from high-risk patterns (Gleason grade 4, which are poorly formed or fused together). Because the disease shows high variability in appearance, inter-observer agreement among pathologists is poor, especially for complex glandular patterns.
Automated AI approaches for prostate cancer grading have largely relied on weakly supervised learning, where models are trained using only slide-level diagnoses rather than precise gland-level annotations. This limits their ability to capture the fine architectural details that define cancer grade and tumor burden.
Accurate gland segmentation -- the ability to precisely outline individual glands in a tissue image -- is a critical foundation for downstream tasks like Gleason grading, biomarker discovery, and survival analysis. Yet a reliable automated segmentation model for prostate cancer was lacking before this work.
The researchers proposed a three-stage training pipeline built on a dual-path Swin Transformer UNet architecture. Swin Transformers are a type of deep learning model that processes images in hierarchical, overlapping windows, making them well-suited to capturing both fine and coarse tissue features simultaneously.
Three datasets were used: (1) SICAPv2, a public dataset of 155 whole-mount biopsy slides with high-quality pixel-level Gleason annotations; (2) the PANDA Challenge, a large-scale dataset of over 11,000 prostate biopsies from Netherlands and Sweden; and (3) an in-house NCI dataset of 150 whole-mount slides with patch-level tumor labels, contributing over 273,000 image patches.
All WSIs were preprocessed into uniform 512x512 pixel patches at 10x magnification using the CLAM pipeline, which automatically segments tissue and removes background. More than 1.6 million patches were generated in total across the three datasets, creating a large and diverse training pool.
The combination of biopsy and surgical whole-mount specimens was intentional, ensuring the model was exposed to the full heterogeneity of benign and cancerous prostate tissue -- an important consideration given how varied prostate cancer can appear under a microscope.
The first training stage used Masked Image Modeling (MIM), a form of self-supervised learning that does not require human annotations. Specifically, the researchers used the SimMIM approach, where random portions of an image are masked and the model is trained to reconstruct the hidden pixels.
This pretraining phase used all available patches -- including unlabeled ones -- to teach the model general features of prostate tissue appearance. By learning to reconstruct masked regions, the encoders develop a rich internal representation of what healthy and cancerous tissue looks like.
Two encoder sizes were pretrained: Swin-Tiny and Swin-Base, each with different patch sizes (4 and 8), enabling the network to capture features at multiple spatial scales. SimMIM was chosen over contrastive self-supervised methods because it showed superior performance when fine-tuned for downstream segmentation tasks.
Directly fine-tuning the pretrained model for segmentation produced limited gains, likely because the features learned during reconstruction differed from those needed for precise gland boundary detection. To address this, the researchers introduced a novel tumor-guided self-distillation step.
Self-distillation involves a teacher model and a student model. The teacher's features -- enriched with patch-level binary tumor or non-tumor labels used as hard attention signals -- guide the student to learn better representations for gland segmentation without requiring full pixel-level annotations.
This step cleverly uses the relatively easy-to-obtain patch-level tumor labels (simply whether a patch contains cancer or not) as a bridge signal. These labels are injected at multiple stages of the Swin Transformer encoder, nudging the model to develop features that are aware of tumor context without overfitting to a simple classification task.
The results confirmed that self-distillation provided the single largest performance boost in the ablation study, improving mean Dice by more than any other single training step -- demonstrating the value of this bridging approach.
The final segmentation model uses a dual-path Swin Transformer UNet structure. After an initial shallow feature extractor captures low-frequency spatial information, the image is split into two parallel Swin Transformer encoder branches with different patch sizes (4 and 8) to capture features at different spatial resolutions simultaneously.
At the end of each transformer stage, features from both branches are fused using a lightweight multi-head attention module followed by a convolution layer. This design is more computationally efficient than prior dual-path architectures while preserving the multi-scale feature integration that improves segmentation accuracy.
The decoder uses deconvolutional layers and residual connections (from the UNETR architecture) rather than pure transformer blocks, reducing computational cost without hurting performance. Additionally, each encoder branch includes a classification head that predicts whether a patch contains tumor tissue.
This classification head serves as a post-processing mechanism: if both heads agree with high confidence that a patch is non-cancerous, the model forces a negative segmentation output, effectively eliminating spurious small segmentation artifacts without any manual post-processing.
The framework achieved a test mean Dice of 0.947 on the PANDA dataset and 0.664 on SICAPv2, outperforming all baseline methods. The mean Dice coefficient measures overlap between predicted and true gland boundaries, with 1.0 being perfect.
In a systematic comparison, the proposed model exceeded classical deep learning approaches including UNet, ResUNet, UNETR, SwinUNETR, and the state-of-the-art nnUNet v2. The base model architecture trained from scratch already surpassed most baselines, and each added training step (MIM pretraining, self-distillation, PANDA training) contributed incrementally to the final result.
The researchers also tested SAM (Segment Anything Model) and MedSAM, two large foundational AI models with impressive zero-shot performance in other domains. Although these models achieved reasonable zero-shot Dice scores of 0.55 and 0.56, fine-tuning them on the prostate dataset did not bring them to state-of-the-art levels -- suggesting that generic foundational models struggle in highly specialized histopathology tasks.
Importantly, the PANDA-trained model was directly tested on the SICAPv2 dataset without any retraining and achieved a zero-shot mean Dice of 0.664, higher than any fine-tuned foundational model, demonstrating strong generalizability across datasets and tissue sources.
The SICAPv2 dataset, while high-quality, comes from a single center (only 95 patients), which limits evaluation of how well the model generalizes to other hospitals, scanners, or patient populations. Multi-center validation will be essential before clinical deployment.
The PANDA segmentation masks used in intermediate training are acknowledged to be noisy and unreliable at the pixel level, which is why PANDA results were not included in the main comparison tables. Semi-automated expert annotation of these large datasets remains a key unresolved challenge in the field.
Expert pathologist review of 50 randomly sampled patches revealed that the SICAPv2 ground truth annotations are somewhat forgiving at gland boundaries, meaning that models producing slightly over-segmented outputs at fine contours may not actually be wrong -- the ground truth itself may undercount fine detail.
Future directions include exploring domain adaptation and progressive fine-tuning strategies to improve foundational models like SAM for specialized histopathology tasks, as well as expanding self-supervised pretraining to additional tissue types and cancer subtypes.
This work presents the first reliable, fully automated framework for prostate gland segmentation in whole-slide pathology images, achieving state-of-the-art accuracy by combining large-scale self-supervised pretraining with a novel tumor-guided distillation strategy.
Accurate gland segmentation enables a wide range of downstream AI tasks that are currently limited by the lack of gland-level features: Gleason grading, tumor burden quantification, cancer biomarker discovery, and survival prediction models that go beyond weakly supervised approximations.
The researchers also released a large-scale whole-mount dataset with patch-level tumor labels to the research community, providing a valuable resource for training and benchmarking future self-supervised and segmentation models in prostate pathology.
By demonstrating that a domain-specific, carefully designed model outperforms large generic foundational models for specialized tasks like prostate gland segmentation, this study supports the value of task-specific AI development in computational pathology rather than relying solely on general-purpose foundation models.