Before a prostate cancer patient can receive radiation therapy, a radiation oncologist must carefully draw boundaries around two critical structures on CT scans: the clinical target volume (CTV) -- the prostate tissue that needs to be irradiated -- and the organs at risk (OARs) such as the bladder, rectum, and femoral heads (hip bones) that must be protected from excessive radiation.
This manual contouring process is time-consuming and subject to substantial variability. Different radiation oncologists draw boundaries differently (inter-observer variability), and even the same physician may draw them differently on different occasions (intra-observer variability). This inconsistency directly affects treatment quality -- overly generous contours expose more healthy tissue to radiation, while overly conservative ones may miss cancer cells.
The main technical challenge is that prostate boundaries are poorly visible on CT images due to low soft tissue contrast. The prostate, bladder, and rectum have similar tissue densities on CT, making them difficult to distinguish, especially where they are in close physical contact. The shapes and sizes of these organs also vary significantly between patients and even change day-to-day as bladder and rectal filling changes.
Automated contouring using deep learning could solve these problems by producing consistent, fast contours that free clinicians from tedious manual work and reduce delays in starting treatment -- particularly valuable in clinics with limited physician time. This study developed and tested a novel deep learning approach for this task.
Convolutional neural networks (CNNs) have been the dominant approach for medical image segmentation for the past decade. Architectures like U-Net use an encoder to extract features at multiple scales and a decoder to reconstruct precise boundaries. CNNs excel at capturing fine local texture and edge information, but they have an inherent limitation: they only look at small local regions of an image at a time, missing long-range spatial relationships between structures that may be separated across the image.
Vision transformers (ViTs) take a fundamentally different approach, borrowed from language processing AI. The input image is divided into patches, which are processed as a sequence. Through self-attention mechanisms, the ViT can relate every patch to every other patch simultaneously, capturing global context across the entire image. This is valuable for anatomy where knowing the overall organ layout helps disambiguate local boundaries.
However, ViTs have their own weakness: they produce coarser outputs and struggle to recover fine spatial detail, particularly at organ boundaries where precision matters most. The natural solution is a hybrid CNN-ViT architecture that uses each component for what it does best: CNN for local detail, ViT for global context. Several such hybrid architectures have been proposed, with TransUNet being an influential early example.
This study designed a novel hybrid architecture with a specific innovation: a custom attention-based fusion module that intelligently combines the CNN and ViT outputs rather than simply concatenating them. The fusion module uses global attention to weight the contributions of each branch and spatial attention to selectively emphasize the most informative combined features.
The study used 104 prostate cancer patients from three radiotherapy centers in Mashhad, Iran, scanned on three different CT scanner models from GE, Siemens, and PNMS. The multi-center, multi-scanner dataset reflects realistic clinical diversity -- a critical design choice because models trained on a single scanner often fail to generalize. Expert contours from a radiation oncologist served as the reference standard for five structures: prostate, bladder, rectum, right femoral head, and left femoral head.
The data was split 70%/10%/20% for training, validation, and testing. Data augmentation was applied during training -- including rotations, flips, crops, zooming, and shearing -- to artificially increase training data diversity and improve the model's robustness to anatomical variation. CT images were preprocessed by cropping to the pelvic region only (reducing computational load) and normalizing intensity values.
Four network configurations were tested: two pure CNNs (VGG16-UNet and ResNet50-UNet), one pure transformer (Swin-UNet), and two hybrid CNN-ViT networks (VGG16-UNet-ViT and ResNet50-UNet-ViT). Transfer learning from ImageNet pre-trained weights was used for the CNN encoders. All five structures were segmented simultaneously in a single forward pass, with a weighted cross-entropy combined with intersection-over-union loss to handle the class imbalance between small and large structures.
Performance was evaluated using Dice Similarity Coefficient (DSC) -- the primary metric measuring volumetric overlap between automated and manual contours -- along with Hausdorff distance (HD95, the 95th percentile of maximum surface distance), average surface distance (ASD), and volume-based error metrics. Statistical significance of improvements was assessed using paired t-tests.
The best-performing model, VGG16-UNet-ViT, achieved Dice scores of: prostate 91.75%, bladder 95.32%, rectum 87.00%, right femoral head 96.30%, and left femoral head 96.34%. These results represent statistically significant improvements over both the pure CNN and pure ViT baselines for all structures (p-values ranging from 0.001 to 0.025).
The pure ViT (Swin-UNet) performed worst overall, achieving only 83.91% DSC for the prostate and struggling with the rectum (80.46%), confirming that transformer-only approaches lose too much spatial resolution for this task. The pure CNN baselines performed substantially better than ViT alone, but still consistently fell behind the hybrid models for all five structures.
The hardest structure to segment was the rectum, where even the best model achieved only 87.00% DSC and had the highest Hausdorff distance (4.46 mm). This reflects the biological reality: the rectum's tubular shape and its proximity to the prostate, combined with highly variable contents, make boundary delineation genuinely difficult. The femoral heads were easiest, consistently exceeding 96% DSC due to their high contrast on CT.
The 7x7 convolutional kernel in the fusion attention module consistently outperformed smaller kernels (3x3 and 5x5) for most structures. The larger receptive field captures more spatial context when combining CNN and ViT features, suggesting that the fusion step itself benefits from looking at a broader neighborhood when deciding how to weight the two information streams.
Compared to prior published work, the proposed VGG16-UNet-ViT achieved the highest reported prostate DSC at 91.75%, surpassing a range of previous deep learning methods. Prior CNN methods on similar datasets achieved prostate DSCs of 82-89%, while the best previously reported hybrid transformer network reached only 84% for the prostate. The improvement is particularly notable because the current study used a more challenging multi-center dataset with multiple scanner types rather than a homogeneous single-center dataset.
One study achieved a higher bladder DSC (97% vs. 95.32%), but it relied on a single scanner and single observer reference standard -- introducing potential bias in the ground truth that inflates apparent performance. The present study's multi-center multi-observer design provides a more rigorous and clinically realistic benchmark.
The rectum DSC of 87% was lower than some prior studies, but the authors note an important confound: those better-performing studies used patients with endorectal balloon insertion, a device that physically stabilizes the rectum during treatment. Without this device, rectal shape varies dramatically between scans, making accurate automated delineation inherently harder. The comparison is therefore not directly applicable to the majority of centers that do not routinely use endorectal balloons.
Automated contouring systems like this one offer concrete clinical benefits: reduced time from CT simulation to treatment start, consistent contours that do not depend on which physician or time of day the contouring is done, and reduced workload for radiation oncologists. The multi-center validation in this study is particularly important because it demonstrates the model works across different scanners -- a prerequisite for broad clinical adoption.
The authors explicitly note that automated contours should be treated as a starting point requiring physician review and modification, not as final treatment contours. Automated contouring is a workflow acceleration and consistency tool, not a replacement for clinical judgment. This is especially important for the rectum and prostate, where errors carry significant clinical consequences (inadequate target coverage or excess rectal dose).
Future work should address several gaps: the dataset size of 104 patients is relatively small, and larger prospective multi-institutional validation is needed. Extension to MRI-based contouring (where prostate boundaries are much clearer than on CT) is a natural next step. Incorporating uncertainty quantification into the model -- flagging regions where the AI is less confident -- could help clinicians prioritize their review effort, focusing manual attention where automated results are least reliable.