Prostate cancer is one of the most common cancers in men worldwide. Many patients undergo radical prostatectomy, a surgery that removes the prostate gland, as their primary treatment.
Despite successful surgery, biochemical recurrence (BCR) occurs in 20 to 40 percent of patients. BCR is detected when prostate-specific antigen (PSA) levels rise again after surgery, signaling that cancer cells are regrowing. BCR is a strong warning sign for future spread of cancer and death.
Identifying which patients are at high risk for BCR before surgery is critical. It allows doctors to plan more aggressive treatments such as chemotherapy, radiotherapy, hormone therapy, or extended lymph node removal for those who need it most.
Current prediction tools rely heavily on Gleason grading, a system for classifying prostate cancer tissue under the microscope. However, Gleason grading was developed in the 1960s and has known limitations, including variability between pathologists and its inability to capture fine-grained tissue details that may predict recurrence.
Researchers collected 1,585 whole slide images (WSIs) from prostate needle biopsy samples taken from 317 patients at two hospitals in China. Five WSIs per patient were selected, all containing cancer tissue, to ensure rich coverage of the tumor.
The AI system works in three stages. In stage one, each slide is divided into small image patches that a convolutional neural network (CNN) examines to estimate the probability of recurrence at the patch level. The Inception_v3 architecture was chosen after comparing it against ResNet50, VGG19, and ResNet18.
In stage two, multiple instance learning (MIL) is used to combine all patch predictions into a single summary for each slide. Two pipelines called PALHI (Patch Likelihood Histogram) and BoW (Bag of Words) capture the distribution and relative importance of patches across the full slide.
In stage three, WSI-level features from all five slides per patient are pooled together and combined with clinical information such as age, PSA value, and Gleason scores. Several machine learning classifiers including multilayer perceptron (MLP), logistic regression, support vector machine, and random forest are then used to predict whether a patient will experience BCR.
Each slide is broken into 512 by 512 pixel patches at a standardized resolution. Background areas are filtered out, and color normalization is applied using the Vahadane method to ensure consistency across slides from different scanners and hospitals.
Transfer learning was used to give the model a head start, initializing it with weights learned from the large ImageNet image database before fine-tuning it on the prostate biopsy data. A cosine decay learning rate schedule helped the model train stably.
The TF-IDF (term frequency-inverse document frequency) technique, borrowed from information retrieval, assigns higher importance to tissue patterns that appear frequently in one slide but are rare across all slides, helping identify distinctive cancer features.
Feature selection used LASSO (Least Absolute Shrinkage and Selection Operator) regression to eliminate redundant or unimportant variables, and Pearson correlation analysis removed highly overlapping features before final model training.
At the patch level, the Inception_v3 model achieved an AUC of 0.803 on the independent testing cohort, meaning it correctly distinguished high-risk from low-risk tissue patches most of the time.
Performance improved at each higher level of the system. At the slide level, the best classifier reached an AUC of 0.848. At the full patient level combining pathology images with clinical features, the best model (MLP classifier) reached an AUC of 0.911 with a 95 percent confidence interval of 0.840 to 0.982.
Using more slides per patient consistently improved prediction accuracy. When all five slides were included, models achieved the highest AUC values compared to using only one or two slides, confirming that broader tissue sampling captures more relevant information.
Decision Curve Analysis (DCA) confirmed that the system provides real clinical benefit, meaning that using it to guide treatment decisions would lead to better outcomes compared to treating all patients the same or treating none.
Gradient-weighted Class Activation Mapping (Grad-CAM) was used to visualize which parts of the tissue images the AI focused on when predicting recurrence. This revealed that the AI paid attention to regions consistent with what experienced pathologists know to be aggressive tissue patterns.
Areas the AI associated with higher BCR risk included the cribriform arrangement (a sieve-like tissue pattern), suspicious vascular gap infiltration, neural invasion, necrosis, and sharp-angle glandular structures, all of which align with established markers of aggressive prostate cancer.
Dimensionality reduction using t-SNE (t-distributed stochastic neighbor embedding) showed that patient-level features formed clearly separated clusters for BCR and non-BCR cases. Average pooling of slide features produced better cluster separation than maximum pooling, resulting in a higher AUC.
Interestingly, clinical stage information (T-stage) was deliberately excluded from the final model. T-stage is often only confirmed after surgery and adding it did not meaningfully improve performance, so the system was designed to work with information available before the operation.
The system is designed to work as a preoperative tool, meaning it can flag high-risk patients before radical prostatectomy takes place. This timing matters because treatment decisions about adjuvant therapies, lymph node removal, and hormonal treatment need to be made before or during surgery.
For patients identified as high risk, the system suggests considering more aggressive approaches such as additional chemotherapy, radiotherapy, hormone therapy, immunotherapy, or prophylactic extended pelvic lymphadenectomy.
A key advantage of the system is that it can accept any number of biopsy slides as input. While performance is best with five slides, it remains useful even with fewer, making it adaptable to different clinical settings where slide availability may vary.
The study has limitations including the use of a retrospective dataset and a requirement that all patients had at least five biopsy cores containing cancer tissue, which may limit generalizability to patients with less extensive biopsies. A prospective validation study would be the next step.
This study demonstrates that a deep learning system trained on prostate biopsy slide images can reliably predict the risk of biochemical recurrence before surgery, achieving an AUC of 0.911 in an independent testing cohort.
The system goes beyond traditional Gleason grading by capturing subtle microscopic tissue features that conventional pathology methods miss. Combining these AI-derived features with standard clinical data produced the best overall performance.
The interpretability of the model, validated through Grad-CAM visualizations, shows that the AI is learning biologically meaningful patterns rather than random artifacts, increasing trust in its clinical applicability.
With further prospective validation and expansion to larger and more diverse patient populations, this type of system has the potential to become a standard component of preoperative prostate cancer assessment, helping doctors deliver more precise and personalized care.