Prostate cancer (PCa) is one of the most frequently diagnosed cancers in men worldwide, with an estimated 300,000 new cases projected in 2024 alone. While many patients are successfully treated, biochemical recurrence affects 20 to 40 percent of them, meaning cancer returns as indicated by rising levels of prostate-specific antigen (PSA) in the blood after treatment.
The challenge is that PSA, the main tool used to detect recurrence, lacks specificity. A rising PSA does not always mean the cancer has returned in a dangerous way, and PSA cannot distinguish between aggressive and slow-growing tumors. This leads to high rates of false positives and uncertainty in clinical decision-making.
Current prognosis tools rely on three clinical variables: serum PSA levels, the Gleason score (a measure of how abnormal the tumor tissue looks under the microscope), and tumor staging (how far the cancer has spread). While these are useful, they leave significant gaps in predictive accuracy.
This study asked whether adding gene expression data, measuring how active specific genes are in a patient's tumor, could improve machine learning models that predict recurrence. The hope is that certain genes could serve as molecular biomarkers, offering earlier and more precise prognostic information before treatment even begins.
The study used a dataset of 489 prostate cancer patients from The Cancer Genome Atlas (TCGA), a large publicly available database of cancer genomic and clinical data. After removing patients with missing information, 419 patients were included, of whom 85 had cancer recurrence events and 334 did not.
In addition to clinical variables (PSA, Gleason score, tumor stage), the researchers measured the expression levels of seven genes using RNA sequencing: KLK3, AR, GSTM3, NETO2, HPN, PRUNE2, and FOLH1. These genes were selected because prior research linked them to prostate cancer biology and outcomes.
Three supervised machine learning algorithms were compared: Naive Bayes (NB), which uses probability calculations; Support Vector Machine (SVM), which finds optimal decision boundaries in multidimensional data; and Artificial Neural Network (ANN), which mimics brain-like learning through interconnected nodes. All three are widely used for medical classification tasks.
To address the imbalance between the small number of recurrence cases and the larger non-recurrence group, random subsampling was used to equalize class sizes. Each model was trained and tested 50 times on different random subsets of the data to ensure robust and reproducible results.
Feature selection was performed using a chi-square test to rank each variable by how strongly it was associated with the recurrence outcome. This step reduces the number of inputs to the model, avoiding redundant or weakly informative data that could reduce accuracy or cause the model to overfit the training data.
As expected, the top three features were the established clinical variables: tumor staging, Gleason score, and preoperative PSA level. These confirmed the relevance of standard clinical tools in predicting recurrence.
Among the genes, the most informative were NETO2, followed by HPN and AR (androgen receptor) tied for the same score, and then KLK3. The remaining genes (GSTM3, PRUNE2, FOLH1) had scores below 0.5 and were not considered relevant for this dataset.
Six predictor groups were tested, ranging from clinical variables alone (Group 1) to combinations that progressively added the selected genes (Groups 2 through 6). This systematic approach allowed direct measurement of how much each gene added to prediction accuracy.
All three models exceeded 60 percent accuracy and 65 percent sensitivity across all predictor groups, meaning they correctly identified more than two-thirds of patients who experienced recurrence. The ANN was the most consistent, showing similar performance across all six predictor groups, suggesting it is the most stable for this type of data.
The SVM showed the strongest performance in precision and specificity, particularly when gene expression data was included. Without genes (Group 1), SVM specificity was only 46 percent. Including NETO2 and HPN (Group 3) raised it to 68.5 percent, an improvement of 22.5 percentage points, while keeping sensitivity above 70 percent. This balance is clinically important: high specificity reduces unnecessary follow-up treatment, while high sensitivity avoids missing true recurrences.
The Naive Bayes models consistently achieved high sensitivity (above 80 percent in five of six groups) but had poor specificity, often below 50 percent. This means NB rarely missed a true recurrence case but also frequently flagged patients who were not at real risk, generating a high rate of false alarms.
The combination of all four selected genes together with clinical variables (Group 6) achieved the highest accuracy for SVM at 70 percent, suggesting that a more complete molecular panel may offer incrementally better overall classification performance.
NETO2 was the most consistently informative gene across all models and predictor groups. Research in other cancer types has linked NETO2 to invasion and metastasis through a cell signaling pathway called PI3K/AKT/NF-KB, and prior studies have also found it to be dysregulated in prostate, breast, and colorectal cancer.
The HPN gene encodes a protein associated with tumor invasion and spread to other tissues. Several studies have connected HPN variants to prostate cancer risk, and its consistent appearance in high-performing predictor groups in this study supports its role as a marker of aggressive disease.
The AR gene encodes the androgen receptor, a protein that drives prostate cancer growth by responding to male sex hormones. It is one of the most studied genes in prostate cancer biology, and its inclusion in better-performing models aligns with its well-established role in promoting cancer progression and treatment resistance.
KLK3, which encodes the protein PSA itself, appeared as a molecular biomarker candidate. While PSA protein levels in blood are already used clinically, the expression level of the gene in tumor tissue carries distinct prognostic information not captured by the simple blood test, potentially enabling more nuanced assessment of recurrence risk.
The core finding of this study is that incorporating tumor gene expression data alongside standard clinical variables meaningfully improves the ability to predict prostate cancer recurrence. This matters because earlier and more accurate prediction allows doctors to customize treatment plans before and after primary therapy.
A key limitation is the relatively high standard deviations in model performance metrics, particularly for specificity. For example, SVM specificity in Group 3 was 68.50 plus or minus 16.41 percent, meaning there was substantial variability across the 50 simulation runs. This suggests the models are sensitive to which patients happen to end up in the training versus testing set, which is a sign that a larger dataset is needed to stabilize the results.
The class imbalance in the dataset, with only 85 recurrence cases out of 419 patients, is another challenge. Even with subsampling to equalize classes, the small number of true recurrence cases limits how well the models can learn the patterns that distinguish recurrent from non-recurrent disease. A larger, more balanced prospective dataset would substantially strengthen confidence in these models.
Despite these limitations, the study demonstrates a practical path forward: combining RNA sequencing data from tumor biopsies with clinical variables, feeding both into machine learning classifiers, and using the output to risk-stratify patients before they begin treatment. This approach is feasible with existing technology and could be implemented in clinical research settings.
This study confirms that machine learning models combining standard clinical variables with gene expression data can predict prostate cancer recurrence with moderate but meaningful accuracy, outperforming models using clinical data alone.
The four genes identified as candidates for a molecular biomarker panel, NETO2, HPN, AR, and KLK3, each have biological rationale supporting their role in cancer progression. Including them in future prognostic tests could help identify high-risk patients earlier, before primary treatment even begins.
The SVM algorithm was most improved by adding gene expression data, while the ANN was most stable across different combinations. Each algorithm has clinical trade-offs: high sensitivity models catch more true recurrences but generate more false alarms, while high specificity models are more precise but may miss some true cases. The right balance depends on the clinical goal.
The next steps for this research include validating the model in independent patient cohorts from different institutions, expanding the gene panel, and exploring whether integrating multi-omic data such as protein expression, microRNA profiles, and epigenetic markers could further improve predictive accuracy for personalized prostate cancer management.