Multiparametric MRI (mpMRI) has transformed prostate cancer diagnosis by enabling direct visualization of suspicious lesions before biopsy. The Prostate Imaging Reporting and Data System (PI-RADS) provides a standardized five-point scale -- from PI-RADS 1 (very unlikely to be significant cancer) to PI-RADS 5 (very likely significant cancer) -- to guide biopsy decisions based on MRI findings.
Despite this standardized framework, PI-RADS scoring is highly subjective. Interreader agreement between radiologists has been reported at less than 50% in published studies, and intrareader agreement (the same radiologist reading the same scan twice) falls in the range of 60 to 74%. In one multicenter study, experienced radiologists disagreed with their own diagnoses 15 to 40% of the time when re-reading the same scan one month later.
The practical consequence of this variability is significant: because PI-RADS scores routinely determine whether a patient undergoes prostate biopsy, inconsistent scoring leads to inconsistent biopsy decisions. Cancer detection rates from MRI-guided biopsies have varied by as much as 40% for PI-RADS 5 lesions across different radiologists and institutions, undermining the clinical utility of the scoring system.
An AI-based PI-RADS scoring system could address variability by producing consistent, reproducible assessments that do not fluctuate based on reader fatigue, experience level, or institutional practice patterns. This study, conducted at the National Cancer Institute (NIH), aimed to develop and evaluate a deep learning model capable of assigning PI-RADS scores to lesions already identified and segmented by a radiologist.
The study used data from three independent cohorts totaling 687 patients and 1,034 detected lesions. Two cohorts came from the NIH Clinical Center (patients who underwent mpMRI before radical prostatectomy or for evaluation of suspected prostate cancer), and a third from the publicly available PROSTATEx challenge dataset. Using three distinct datasets was designed to expose the neural network to diverse patient populations and scanner configurations, reducing overfitting to a single center's imaging characteristics.
All lesions were manually segmented by an expert radiologist with more than 10 years of prostate MRI experience (more than 1,200 MRIs per year). Bounding boxes were placed around each lesion segmentation on the T2-weighted, ADC (apparent diffusion coefficient), and high-b-value diffusion images, and the three image channels were stacked and saved as JPEG images for use as model inputs.
A ResNet-34 convolutional neural network (CNN) was trained using the fastai deep learning library, with weights pre-initialized from a model trained on ImageNet (transfer learning). The model was trained to classify each lesion slice into one of four PI-RADS categories (2 through 5). Standard data augmentation techniques -- including random flips, rotations, and a strategy called mixup (blending pairs of training images) -- were applied to increase robustness.
The combined dataset was split into training (70%), validation (20%), and test (10%) sets on the patient level, preventing data leakage between sets. During inference, the CNN assigned a PI-RADS score to each individual slice within a lesion, and the softmax probability outputs were averaged across all slices to derive a final per-lesion PI-RADS classification. The dataset contained slices distributed across all PI-RADS categories, with PI-RADS 5 slices being the most common (40% of total slices).
The AI system achieved an overall lesion-level agreement with the expert radiologist of 58%, with a kappa score of 0.40 (95% CI: 0.32-0.48) -- interpreted as moderate agreement. Notably, this falls within the range of published interreader and intrareader agreement figures for human radiologists, which have been reported as low as kappa 0.24 in multicenter studies.
Agreement varied substantially by PI-RADS category: it was lowest for PI-RADS 2 lesions (6%) and highest for PI-RADS 5 lesions (80%). When a one-score tolerance was applied -- counting the AI as correct if its score was within one PI-RADS category of the radiologist's score -- the correct classification rate improved to 86%.
The AI system showed a tendency toward upgrading lesions: 28% of lesions were assigned a higher PI-RADS score by the AI than by the radiologist, compared with only 15% that were downgraded. This upgrading bias was most pronounced for PI-RADS 3 lesions, where the AI upgraded 44% while downgrading only 4%. This pattern suggests the system may be appropriately cautious about potentially significant lesions.
The interreader comparison experiment confirmed that human PI-RADS variability is substantial: two experienced radiologists agreed on the exact PI-RADS score in only 50% of 50 cases from the PROSTATEx dataset, with a kappa of 0.34 -- lower than the AI system's performance against the expert radiologist. This contextualizes the AI's moderate agreement as comparable to human-to-human variability.
The most clinically meaningful test of the AI system was whether its PI-RADS scores predicted clinically significant cancer on biopsy as effectively as the expert radiologist's scores. Of the 307 lesions in the validation and test sets, 86 had undergone MRI/ultrasound fusion-guided targeted biopsy within 6 months of MRI acquisition at the NIH.
For the radiologist-assigned PI-RADS scores, clinically significant cancer (Gleason grade 3+4 or higher) was detected in 0% of biopsied PI-RADS 2 lesions, 50% of PI-RADS 3, 40% of PI-RADS 4, and 79% of PI-RADS 5 lesions. For the AI-assigned scores, the corresponding rates were 0%, 40%, 39%, and 85%.
The differences between AI and radiologist cancer detection rates for each PI-RADS category were not statistically significant (p = 0.59 for PI-RADS 3, p = 0.36 for PI-RADS 4, p = 0.47 for PI-RADS 5). This is the study's key finding: despite only moderate agreement in the absolute PI-RADS scores assigned, the AI's scores mapped to cancer risk at the biopsy level as effectively as the expert radiologist's scores.
The authors explain this apparent paradox by noting that clinically significant cancer rates are similar between adjacent PI-RADS categories -- for example, PI-RADS 3 and PI-RADS 4 lesions have only about a 10% difference in cancer rate. Therefore, a one-category disagreement between the AI and radiologist in score may not translate to a meaningful difference in clinical decision-making about whether to perform biopsy.
The central conclusion is that PI-RADS scoring agreement should not be evaluated in isolation from clinical outcomes. An AI system with kappa of 0.40 would traditionally be considered a modest performer, but when its scores predict cancer on biopsy as well as an expert radiologist's scores, it provides equivalent clinical value. The study frames AI as a tool for standardizing PI-RADS interpretations, not necessarily replicating every scoring decision of a single expert.
A key limitation is that the AI requires manual lesion segmentation as its first step -- the system classifies already-identified lesions rather than detecting them de novo. This means the AI cannot replace the radiologist's role in lesion detection; it assists only with the subsequent scoring step. A future fully automated system would need to combine lesion detection and classification in a single pipeline.
The study used PI-RADS version 2 rather than the newer version 2.1, which was released during the study period. Scoring criteria differ between versions, particularly for transition zone lesions, so performance with the updated guidelines may differ. Additionally, pathologic validation was available for only 46% of in-house lesions, and was based on targeted biopsy rather than whole-mount pathology from radical prostatectomy, which may miss some cancer.
The tendency of the AI to upgrade PI-RADS 3 lesions at a high rate (44%) has mixed clinical implications. Upgrading PI-RADS 3 to PI-RADS 4 lesions could trigger biopsies in some patients who might not otherwise be recommended for one, potentially increasing cancer detection while also increasing unnecessary procedures in men without clinically significant cancer.
This study demonstrates that a deep learning model trained on multiparametric prostate MRI data from three independent cohorts can assign PI-RADS scores with moderate agreement to an expert radiologist, at a level comparable to human interreader variability. Crucially, the AI's scores were equally predictive of clinically significant cancer on targeted biopsy, establishing clinical equivalence on the most important outcome measure.
The AI was developed at the NIH using data from expert-level radiologists at a high-volume institution, and the inclusion of the public PROSTATEx dataset introduces some external generalizability. The model code has been made publicly available, enabling other research groups to build on this work and adapt the model to their own institutional data.
Future development should focus on combining lesion detection with PI-RADS classification in a single end-to-end pipeline, validating performance against PI-RADS v2.1 criteria, and testing the model in lower-volume centers where radiologist expertise is more variable. Prospective clinical trials where AI-assisted and unassisted PI-RADS scoring are compared head-to-head on biopsy outcomes would provide the highest level of evidence for clinical adoption.