Prostate cancer is the second most commonly diagnosed cancer in men and the fifth leading cause of cancer-related deaths worldwide, accounting for roughly 15% of all male cancers globally. It is the most prevalent cancer among men in 112 countries, with an average age of presentation around 66 years. The annual caseload is projected to roughly double, from 1.4 million new cases in 2020 to 2.9 million by 2040. This rising incidence places enormous pressure on diagnostic workflows.
Current diagnostic bottlenecks: Screening relies on biopsy and analysis of hematoxylin and eosin (H&E)-stained tissue sections, with pathologists assigning Gleason scores to evaluate cancer severity. Each patient typically requires about 12 biopsies, generating millions of samples annually and creating a substantial workload for pathologists worldwide. The Gleason grading system itself is subjective, with significant interobserver and intraobserver variability across pathologists. This subjectivity leads to inconsistent diagnoses and treatment recommendations.
What this report covers: This technical report from the University of South Florida consolidates the current landscape of AI applications in prostate cancer detection and management. It covers AI-driven Gleason scoring, non-invasive biomarker-based screening (particularly PSA integration), MRI-based tumor delineation, multi-omics approaches, and prognostic modeling. The authors also introduce fundamental deep learning concepts to help clinicians understand the technology underpinning these tools.
The report examines specific architectures including fully convolutional networks (FCNs) with U-Net and ResNeXt50, the clinically validated DeepDx Prostate system, and entries from the Prostate Cancer Grade Assessment (PANDA) challenge. It also addresses AI models that integrate PSA data, MRI-based radiomics, and multi-omics datasets for a more comprehensive diagnostic and prognostic pipeline.
AI algorithms for Gleason scoring follow a multistep pipeline that begins with digitizing histopathological slides into high-resolution whole-slide images (WSIs). Preprocessing is critical for ensuring consistency across different laboratory conditions and reducing "batch effects." A key technique here is stain normalization, which compensates for differences in staining protocols across institutions so the model performs consistently regardless of where or how slides were prepared. Data augmentation, including rotations, flips, and random cropping, further improves model robustness by simulating natural variations in how slides appear.
Feature extraction with CNNs and transformers: After preprocessing, convolutional neural networks (CNNs) or transformer-based architectures are trained on labeled subsets to recognize glandular morphologies, nuclear features of malignant cells, and other tissue-level patterns. The trained algorithm then segments and classifies tissue regions into benign or malignant areas and further distinguishes between individual Gleason patterns (grades 1 through 5, though grades below 6 are rarely reported clinically).
Advanced architectures: Models such as U-Net-based FCNs and ResNeXt50-enhanced classifiers use hierarchical feature extraction to detect subtle architectural variants in tumor growth patterns. These systems not only automate scoring but also quantify tumor heterogeneity, producing quantitative metrics that assist pathologists in decision-making. Ensemble learning pools predictions from multiple AI algorithms into a "majority vote" to improve accuracy and robustness. Self-supervised techniques allow algorithms to learn image features for classification without requiring manual labels, reducing the labeling burden and mitigating interobserver variability in the training dataset itself.
The paper includes a comprehensive glossary (Table 1) defining 20+ AI concepts relevant to prostate cancer modeling, covering supervised, semi-supervised, weakly supervised, and self-supervised learning paradigms, as well as techniques such as transfer learning, multiple-instance learning, active learning, data augmentation, cross-validation, and hyperparameter optimization.
U-Net with ResNeXt50: This architecture combines U-Net's encoder-decoder design with ResNeXt50's grouped convolutions. The encoder compresses the input image to capture essential hierarchical features, while the decoder reconstructs these features into high-resolution, pixel-level segmentation maps. U-Net's skip connections link encoder layers to corresponding decoder layers, preserving fine-grained spatial details while integrating higher-order contextual information. ResNeXt50 enhances this by splitting input channels into smaller groups processed independently, improving feature extraction efficiency while remaining computationally feasible.
Atrous Spatial Pyramid Pooling (ASPP): A key addition to this model is ASPP, placed after the encoder-decoder bottleneck. ASPP uses atrous (dilated) convolutions at multiple dilation rates, allowing the model to capture contextual information at multiple spatial scales simultaneously. This means it can analyze both fine cellular-level details and broader glandular organization patterns, then combine all of this multi-scale information into a single output. This capability is especially important for differentiating closely related Gleason patterns such as grades 3 and 4, where subtle architectural differences can directly influence treatment decisions. The model also employs ensemble distillation: five models trained on different data subsets transfer their aggregate knowledge to a single "student" network, improving generalizability and reducing training time.
DeepDx Prostate: This clinically validated system combines a deep neural network with pixel-level segmentation to analyze prostate core needle biopsies. It processes WSIs in two steps. First, patch-level segmentation divides each WSI into fixed-size patches analyzed individually across five categories (non-cancerous through Gleason pattern 5). Second, slide-level evaluation aggregates these patch results into a single heatmap, where the proportion of each Gleason pattern determines the final score. DeepDx Prostate was trained on 1,133 annotated WSIs and validated on 700 cases. It achieved a kappa of 0.907 with expert uropathologists and a tumor quantification correlation coefficient (R) of 0.97 with pathologist-measured tumor lengths, compared to R = 0.90 for original hospital diagnoses.
PANDA Challenge: The 2020 Prostate Cancer Grade Assessment challenge, organized by Radboud University Medical Center and Karolinska Institute, provided 10,616 development biopsies (5,160 from Radboud, 5,456 from Karolinska) plus external validation sets of 741 US and 330 European cases. Top-performing entries achieved quadratic-weighted kappa scores of 0.862 and 0.868 on external validation, demonstrating the generalizability of AI systems trained on large, heterogeneous datasets.
Supervised vs. semi-supervised learning: DeepDx Prostate relies on supervised learning, using pathologist-labeled datasets to achieve high accuracy and concordance. However, creating large, fully labeled datasets is extremely labor-intensive; labeling thousands of WSIs with precise Gleason scores requires multiple pathologists to reach consensus, making this a resource-heavy process. In contrast, the U-Net/ResNeXt50 model uses semi-supervised labeling, combining a smaller set of labeled data with a larger pool of unlabeled data. The algorithm learns underlying patterns from the labeled samples and generalizes to the unlabeled data, reducing dependency on fully annotated datasets while still achieving high performance.
Hard-example mining: This technique optimizes semi-supervised learning by identifying the most challenging cases where model predictions deviate significantly from expert annotations. These difficult examples are flagged and prioritized during training, forcing the model to improve in its weakest areas. By focusing on subtle and ambiguous patterns in histological images, hard-example mining makes the model more robust at handling borderline cases that are most likely to cause diagnostic disagreement.
Transfer learning and cross-domain adaptation: Transfer learning expedites training by leveraging knowledge from pre-trained models. Generic datasets such as ImageNet, Microsoft COCO, and CIFAR, which contain millions of annotated images across hundreds of categories, serve as starting points. These pre-trained models already understand basic visual patterns (edges, shapes, textures), giving them a substantial advantage when applied to prostate tissue WSIs. Cross-domain transfer learning takes this further by adapting models trained on one cancer type (e.g., breast cancer WSIs) to another (e.g., prostate cancer WSIs), bridging dataset gaps across different malignancies.
DeepDx Prostate's architecture details: Beyond its U-Net-like pipeline, DeepDx Prostate uses DeepLab v3+, a neural network designed for detailed image segmentation. It also incorporates non-local attention mechanisms that help the model understand relationships between different parts of the image, improving its ability to analyze complex tissue structures and long-range spatial dependencies within histopathological samples.
AI vs. pathologist agreement: DeepDx Prostate achieved a kappa of 0.713 for Gleason grading and 0.922 for overall concordance with expert pathologists, while original pathology reports showed kappa values of 0.619 for grading and 0.873 for concordance. AI-assisted grading improved general pathologist concordance with expert Gleason scores from kappa = 0.876 (manual) to kappa = 0.925, while also reducing slide examination time by 34%. The FCN with U-Net/ResNeXt50 architecture achieved quadratic-weighted kappa values of 0.92, 0.96, and 0.93 on three test sets, compared to 0.65-0.91 for individual pathologists (both uropathologists and general pathologists).
PSA-integrated AI models: Traditional PSA testing has well-known limitations. Elevated PSA levels can indicate prostate cancer, benign prostatic hyperplasia (BPH), or prostatitis, making PSA alone insufficient for diagnosis. Perera et al. (2021) developed a dense neural network with four fully connected layers trained on total PSA, free PSA, free-to-total PSA ratio, and patient age. The model used stochastic gradient descent for optimization, dropout regularization to prevent overfitting, and power transformations for feature normalization.
The PSA-integrated AI model achieved an AUC of 0.72 on the test dataset, outperforming PSA alone (AUC = 0.63), free PSA (AUC = 0.50), and age alone (AUC = 0.52). At the traditional PSA threshold of 3.0, sensitivity is only 32.2% with specificity of 86.7%. When the AI model was set to match that same 86.7% specificity, its sensitivity improved to 46.4%. At a sensitivity threshold of 80%, the model achieved a specificity of 45.3%, meaningfully reducing unnecessary biopsies and false positives compared to traditional PSA cutoffs.
These results demonstrate the value of combining multiple biomarkers through non-linear decision-making. The AI model's ability to capture complex, non-linear relationships between PSA variants, age, and cancer risk provides a more refined tool for risk stratification than any single biomarker threshold can offer.
Tumor margin precision: Multiparametric prostate MRI (mpMRI) provides critical insights into tumor aggressiveness and margins, but conventional interpretation faces significant challenges in estimating the full extent of prostate cancer. Inaccurate tumor margin estimation can lead to undertreatment with tumor recurrence or unnecessary excision of benign tissue. Mota et al. (2024) demonstrated that AI-derived tumor boundaries achieved substantially higher balanced accuracy (84.7%) compared to standard-of-care physician-delineated contours (67.2%). Sensitivity was even more striking: 97.4% for the AI-assisted method vs. 38.2% for standard-of-care.
Negative margin rates and clinical impact: AI-assisted tumor contouring led to a dramatic increase in negative margin rates, from 1.6% with standard-of-care to 72.8% with AI assistance. This suggests enormous potential for enhancing the precision of focal therapy while reducing overtreatment of non-cancerous tissue. AI-assisted contours altered physician decision-making in 28% of cases, shifting treatment recommendations toward more targeted interventions such as focal therapy and reducing the use of radical prostatectomy in select patients.
Radiomics and quantitative imaging biomarkers: Radiomics extracts quantitative features from biomedical images, called quantitative imaging biomarkers (QIBs), to characterize tumors beyond what visual assessment can reveal. Texture analysis techniques such as the gray-level co-occurrence matrix capture tumor heterogeneity. Machine learning enhances radiomics accuracy for distinguishing benign from malignant lesions and predicting tumor progression. QIBs have also shown potential in monitoring immunotherapy responses, revealing imaging features linked to immune-activated tumor microenvironments.
The report emphasizes that AI-enhanced MRI analysis can integrate with histopathology-based and PSA-based AI models to provide a multi-modal assessment. AI-driven MRI may help guide targeted biopsy strategies by identifying higher-risk tumor regions that might otherwise be overlooked, and the combination of QIBs with liquid profiling methods could facilitate more personalized management of prostate cancer.
Molecular subtyping: Computational analyses of genomic, epigenetic, and tumor microenvironment data have identified distinct molecular subtypes of prostate cancer defined by ETS fusion genes, SPOP mutations, and immune-related phenotypes, each carrying different prognostic implications. Machine learning approaches such as unsupervised clustering and dimensionality reduction techniques are used to classify these subtypes. The PAM50 classifier, originally developed for breast cancer using a 50-gene signature, has been adapted for prostate cancer. Ge et al. found that luminal B subtypes of prostate cancer show higher response rates to androgen deprivation therapy (ADT) compared to luminal A and basal subtypes, despite having an overall poorer prognosis.
Multi-omics integration: Integrating metabolomics, transcriptomics, lipidomics, and genomics provides a richer picture of prostate cancer biology. In metastatic disease, machine learning models combining genomic alterations and lipidomic profiles improved prediction of clinical outcomes. Multi-feature classifiers achieved AUC scores of 0.751 for predicting metastatic hormone-sensitive prostate cancer (mHSPC) survival and 0.638 for predicting androgen deprivation therapy failure. Metabolomics tools have also uncovered altered molecular pathways, such as sphingosine-1-phosphate receptor signaling, revealing loss of tumor suppressor gene downstream signaling that could be targeted therapeutically.
AI prognostic models: Physicians are notably inaccurate in estimating patient prognosis. Studies show that only 20% of prognostic predictions fall within 33% of actual survival, with physicians overestimating survival by a factor of 5.3 on average. For prostate cancer specifically, clinician mortality estimates were more than fivefold higher than model-based predictions, potentially leading to overtreatment such as unnecessary radical prostatectomy. Machine learning models such as random survival forests (RSF) and survival trees have demonstrated superior accuracy. RSF models achieved C-index values up to 0.832, outperforming conventional Cox regression and PSA-based predictions for overall survival and cancer-specific survival in metastatic prostate cancer.
These findings collectively illustrate how multi-omics AI can move prostate cancer management toward truly personalized treatment selection, guided by molecular subtype, metabolic profiling, and data-driven survival estimates rather than clinical intuition alone.
Dataset bias and representation: AI models are susceptible to biases introduced by non-representative training datasets, leading to disparities in diagnostic accuracy across diverse populations. Frewing et al. highlighted observer variability in pathologist annotations as a source of bias, emphasizing the need for more inclusive and representative datasets. If training data disproportionately represents certain demographics or institutional practices, the model may systematically underperform on underrepresented groups.
Domain shift: Differences in imaging protocols, staining techniques, and scanning equipment across institutions cause domain shift bias, which is one of the most persistent challenges in computational pathology. This was demonstrated clearly in the PANDA challenge, where even the best-performing models struggled with diagnosing benign cases in external datasets due to dataset-specific domain shifts. The report stresses the need for universal stain normalization protocols and standardized imaging pipelines to mitigate this issue. Without such standardization, models that perform well in controlled, single-center settings may falter when deployed in diverse real-world clinical environments.
Generalization gap: Small and heterogeneous datasets compound the domain shift problem, constraining model robustness. Models may achieve excellent performance on validation sets that resemble their training data but struggle to generalize to truly external populations. The supervised learning approach used by systems such as DeepDx Prostate requires enormous labeling effort, limiting scalability, while semi-supervised methods trade some labeling burden for potentially lower ceiling accuracy in certain edge cases.
Future directions: The report identifies several priorities for advancing AI in prostate cancer diagnostics. These include developing larger and more diverse multi-institutional training datasets, establishing standardized preprocessing and stain normalization protocols, creating regulatory frameworks for clinical AI deployment, and building multi-modal pipelines that integrate histopathology, PSA, MRI, and multi-omics data into unified decision-support systems. The combination of AI-driven MRI for targeted biopsy guidance with histopathology-based grading and molecular subtyping could ultimately enable a comprehensive, individualized approach to prostate cancer management.