Prostate cancer is the second most common cancer in men globally. Effective management -- from initial staging to radiation therapy planning to MRI-guided biopsy -- requires accurately outlining the prostate gland and its internal zones on MRI scans. This process, called segmentation, is currently performed manually by radiologists tracing boundaries slice by slice, a task that is time-consuming and known to vary significantly between different readers.
Over 70% of prostate tumors arise in the peripheral zone (PZ) -- a distinct anatomical region within the prostate gland. Accurately mapping not just the outer gland boundary but also the peripheral zone boundary provides critical spatial context for tumor localization, biopsy targeting, and focal therapy planning. Most automated segmentation models address only the whole gland and neglect the peripheral zone.
Deep learning has shown great promise for automated segmentation, but a persistent challenge is domain shift: models trained on data from one hospital or scanner tend to perform significantly worse when tested on data from a different site, even for the same imaging task. In prostate MRI, this is compounded by variations in scanner manufacturer, field strength, image resolution, and acquisition protocol across institutions.
This study presents PPZ-SegNet (Prostate and Peripheral Zone Segmentation Network), a deep learning model that simultaneously segments both the whole prostate gland and the peripheral zone from T2-weighted MRI. The network was trained on 150 patients and rigorously evaluated on 283 patients drawn from four entirely different datasets to assess real-world generalizability.
PPZ-SegNet combines two parallel neural networks: a 2D CNN that analyzes individual MRI slices (256 x 256 pixels) and a 3D CNN that processes cropped volumetric patches (128 x 128 x 23 voxels) using 3D convolutions. The 2D network excels at capturing fine detail within each slice, while the 3D network incorporates context across slices to understand the prostate's 3D shape -- combining their strengths in an ensemble.
Both networks use a U-shaped encoder-decoder architecture with residual blocks. The encoder extracts increasingly abstract features from the image, while two separate decoder paths reconstruct segmentations: one path (up-pg) produces the whole prostate gland segmentation, and the other (up-pz) produces the peripheral zone segmentation. Information from the PZ decoder is passed to the prostate gland decoder to ensure the two boundaries are anatomically consistent with each other.
Bayesian hyperparameter optimization was used to automatically determine the optimal network configuration -- including number of convolutional layers, filter counts, kernel sizes, activation functions, and dropout rates. Rather than manually trying different configurations, the Bayesian approach builds a probabilistic model of how different configurations affect accuracy and intelligently selects the most promising configurations to test, finding a better architecture more efficiently than random search.
After the optimal architecture was found, five-fold cross-validation was used to train five separate 2D-3D ensemble models. The final prediction for each new image is determined by majority voting across all five ensembles -- a technique that improves consistency and reduces sensitivity to any single model's errors, especially important for generalizing to unseen datasets.
All data came from publicly available collections on The Cancer Imaging Archive (TCIA). Training used 150 cases from the PROSTATEx Challenge (Train #1), acquired on Siemens 3-Tesla scanners. The four test cohorts were: Test #1 (192 cases from PROSTATEx test set -- same scanner family as training), Test #2 (26 cases from Philips 3-Tesla scanners), Test #3 (15 cases from GE 3-Tesla scanners), and Test #4 (50 cases from the PROMISE12 Challenge, with varied acquisition parameters).
Prostate and peripheral zone boundaries were manually drawn slice by slice by expert radiologists using consensus reads for Train #1 and Tests #1-3. Test #4 used pre-existing annotations provided by the PROMISE12 challenge organizers. The inter-cohort distribution differences were quantified using three statistical metrics: Kullback-Leibler divergence, Wasserstein distance, and Jensen-Shannon divergence -- confirming that Tests #2, #3, and #4 had meaningfully different image intensity distributions from the training set.
All images were preprocessed to a standard resolution of 0.5 x 0.5 x 3 mm by resampling with B-spline interpolation, then resized to a reference plane of 256 x 256 x 23 voxels. Pixel intensities were normalized to the range [0,1] by clipping within 3 standard deviations and rescaling, ensuring consistent input regardless of scanner type.
For whole prostate gland segmentation, PPZ-SegNet achieved a Dice similarity coefficient (DSC) of 0.854 on Test #1 (same scanner family as training), 0.794 on Test #2 (Philips scanners), 0.806 on Test #3 (GE scanners), and 0.622 on Test #4 (PROMISE12). DSC measures overlap between the predicted and reference segmentation, ranging from 0 to 1 where 1 is perfect agreement. Results consistently declined as the image distribution diverged further from the training data.
For peripheral zone segmentation, Dice scores were lower than for the whole gland, reflecting the greater anatomical challenge of delineating this internal structure: 0.664 on Test #1, 0.645 on Test #2, and 0.640 on Test #3. PZ annotations were not available for Test #4, preventing evaluation on that cohort. PZ segmentation is an inherently harder problem due to the less distinct boundary between the PZ and central gland zones on T2-weighted images.
The Hausdorff distance -- measuring the maximum boundary error in millimeters -- was 11.3 mm for Test #1, 12.2 mm for Test #2, 9.5 mm for Test #3, and 21.7 mm for Test #4. The low HD for Test #3 despite its different scanner reflects the fact that this cohort had smaller prostate volumes, reducing the absolute boundary distance even when the boundary was less accurately identified proportionally.
A clear relationship emerged between prostate volume and segmentation accuracy: in three of the four test cohorts, cases with larger prostate glands achieved higher Dice scores. Small glands are harder to segment precisely because subtle spatial errors represent a larger proportion of the total volume. The training set contained predominantly larger glands, meaning the model had less exposure to the appearance of small prostates.
The study tested how changing the training dataset composition affected generalization. When the model was retrained on only 30 cases from Tests #2 and #3 (instead of the original 150 cases from Train #1), performance dropped across nearly all test cohorts -- demonstrating that larger and more diverse training datasets consistently produce better-generalizing models. More training data from PROSTATEx was more valuable than smaller amounts of data from the target scanners.
However, when 50 cases from Train #1 were combined with all cases from Tests #2 and #3 (a mixed training set of 91 cases), performance on Test #4 improved by 7.3% DSC compared to training on PROSTATEx alone. This suggests that including data with a wider range of prostate volumes -- including smaller glands like those in Tests #2 and #3 -- can improve performance on cohorts with small prostates, even when the mixed dataset is smaller overall.
Increasing the number of 2D-3D ensemble networks from 1 to 5 showed minimal effect on Dice score but meaningfully reduced Hausdorff distance in the most challenging cohorts -- from 19.4 mm to 9.5 mm in Test #3 and from 34.4 mm to 21.7 mm in Test #4. This suggests that while ensemble size primarily improves boundary precision rather than overall overlap, the improvement in boundary accuracy is clinically significant for treatment planning applications where contour accuracy directly affects radiation dose delivery.
The sharp performance drop on Test #4 (DSC 0.622 versus 0.854 on Test #1) illustrates a fundamental challenge in medical AI: the same model can perform near-state-of-the-art on one dataset and substantially worse on another. Statistical analysis confirmed that Test #4 images had the most different intensity distribution from training -- their lower image contrast made boundary delineation harder. This demonstrates the importance of testing AI models on diverse external datasets before clinical deployment.
The fact that Test #1 achieved much higher performance than the other test sets despite using an independent set of patients supports one key insight: standardizing imaging protocols to follow PI-RADS guidelines reduces domain shift and produces more consistent AI results. Hospitals that follow standardized T2-weighted imaging protocols will likely see better performance from AI segmentation tools than those with non-standard protocols.
Compared to other published models, PPZ-SegNet's strength is its thorough multi-cohort evaluation -- most competing models report results on internal test sets (typically 12-25% of their training size) rather than large independent cohorts. Testing on 283 fully independent cases provides a more realistic picture of how the model would perform in clinical deployment. The model code is publicly available, enabling other researchers to build on this work.
PPZ-SegNet demonstrates that a Bayesian-optimized 2D-3D ensemble approach can simultaneously segment the prostate gland and peripheral zone from T2-weighted MRI, achieving strong performance on similar datasets and reasonable performance across diverse external cohorts. The dual segmentation capability -- producing both whole gland and zonal anatomy -- fills a gap in existing automated tools and could directly support clinical workflows for biopsy targeting and focal therapy planning.
The study establishes several practical principles for clinical deployment of prostate segmentation AI: larger training datasets improve generalization, ensemble methods reduce boundary errors in challenging cases, and standardizing image acquisition to PI-RADS guidelines minimizes performance degradation at new sites. These findings are broadly applicable to other anatomical segmentation tasks in medical imaging.
Future work is needed to address the persistent performance gap on non-standard imaging datasets. The authors plan to explore additional preprocessing techniques such as histogram matching, domain adaptation methods, and networks that are explicitly designed to be invariant to imaging protocol differences. Expanding the training dataset to include more cases with small prostate volumes and non-standard scanner characteristics would further improve the model's universal applicability.