Stage IV prostate cancer accounts for 15 to 30 percent of all prostate cancer diagnoses. Among these patients, a high-risk or high-volume subgroup faces particularly rapid disease progression, with a median overall survival of only 32.2 months. These patients require more aggressive treatment such as doublet or triplet therapy combining androgen deprivation with new-generation hormonal agents or chemotherapy.
Identifying this high-risk subgroup requires integrating information from multiple imaging and laboratory sources: PSA levels, Gleason score from biopsy, whole-body bone scans, CT scans, and MRI. Clinicians must extract staging data from each source and then apply complex criteria to classify risk, a process that is time-consuming and error-prone when toggling between separate hospital information systems.
Established risk classification criteria include LATITUDE and CHAARTED, which define high-risk metastatic hormone-sensitive prostate cancer based on the number and location of bone metastases, Gleason grade, and presence of visceral disease. Correctly applying these criteria requires integrating multi-modal data and reasoning through specific logical rules, a challenging cognitive task even for experienced urologists.
Traditional NLP tools that automate this data extraction require large, annotated training datasets and substantial infrastructure investment for each specific task. Large language models (LLMs), pre-trained on vast amounts of text, offer a potentially cost-effective alternative that can handle multiple clinical tasks without task-specific retraining.
The study evaluated 350 simulated stage IV prostate cancer patients with complete multi-modality reports covering bone scan, CT or MRI, and biopsy pathology. Starting from a pool of 1,200 simulated combinations, researchers filtered for clinically realistic and reasonably distributed scenarios. This simulation-based approach was necessary because sending real patient reports to third-party AI services raises significant privacy and ethics concerns under institutional review board regulations.
Ten clinical tasks were assessed for each patient: seven information retrieval (IR) tasks including TNM staging, bone metastasis detection, axial and non-axial bone metastatic site counts, and visceral metastasis detection; and three risk assessment (RA) tasks applying the LATITUDE high-risk, CHAARTED high-volume, and Taiwan National Health Insurance (TwNHI) high-risk criteria.
The gold standard for all outcomes was established by three independent expert adjudicators, with inter-rater reliability measured using Cohen's kappa. Kappa values ranged from 0.78 to 0.90 across tasks, confirming strong expert agreement that validates the reference standard.
Four LLMs were tested using zero-shot chain-of-thought (ZS-CoT) prompting, which presents the full multi-modal report followed by structured prompts that instruct the model to reason step-by-step. Each case was queried three separate times per model, and an ensemble voting method aggregated the majority answer across the three rounds to produce a final prediction. Performance was measured using six metrics: accuracy, sensitivity, specificity, positive predictive value, negative predictive value, and F1 score.
In risk assessment tasks, there were major performance differences between the four models. ChatGPT-4-turbo ranked first across all three RA tasks, followed by Claude-3-opus, Gemini Pro 1.0, and ChatGPT-3.5-turbo. ChatGPT-4-turbo achieved ensemble voting accuracy of 90.1% for LATITUDE, 90.7% for CHAARTED, and 91.6% for TwNHI high-risk classification, with intraclass correlation coefficients of 0.86, 0.93, and 0.76 confirming high consistency across repeated queries.
By contrast, ChatGPT-3.5-turbo achieved only 35.1%, 49.3%, and 31.9% accuracy on the same three tasks. This model showed notably high sensitivity and NPV, meaning it correctly identified most true high-risk patients but also flagged large numbers of non-high-risk patients as high-risk, resulting in very low specificity. The performance gap between generations of GPT models was dramatic and statistically significant (odds ratio 16.9 for LATITUDE, 10.1 for CHAARTED, 23.3 for TwNHI).
A striking finding was ChatGPT-4-turbo's strong performance on TwNHI criteria, a Taiwan-specific risk classification documented in traditional Mandarin that is unlikely to appear in the model's training data. ChatGPT-4 matched its performance on the internationally known LATITUDE and CHAARTED criteria, suggesting that its logical reasoning capability generalizes to novel criteria even when given only a brief zero-shot description.
These RA results closely approached human expert performance. The inter-rater kappa values for the three adjudicators on the same tasks were 0.89, 0.84, and 0.89, suggesting that ChatGPT-4-turbo's consistency level is comparable to the natural variability between human physicians applying the same criteria.
In contrast to risk assessment, the four models showed much smaller performance differences on basic information retrieval tasks. TNM staging accuracy ranged from 90.1% to 94.2% across all models, with ICCs above 0.80 for all, indicating that all four models can reliably extract structured staging information from free-text reports. Claude-3-opus and Gemini Pro 1.0 actually matched or slightly exceeded ChatGPT-4-turbo on some IR subtasks.
For bone metastasis detection, the top three models (ChatGPT-4-turbo, Claude-3-opus, Gemini Pro 1.0) all achieved accuracy above 91%, with ChatGPT-3.5-turbo trailing at 82%. Visceral metastasis detection followed a similar pattern: ChatGPT-4-turbo reached 96.8% accuracy, Claude-3-opus 94.8%, Gemini Pro 1.0 84.1%, and ChatGPT-3.5-turbo only 77.1%.
The most difficult IR tasks were counting bone metastatic sites across three categories (0, 1-3, or 4 or more). Non-axial bone metastatic site counting was the weakest task for all models, with accuracies ranging from 64.0% to 81.4%. The challenge is that radiology reports use vague language like multiple, diffuse, or probable that requires interpretation rather than simple keyword extraction, and this ambiguity affects both LLMs and human readers.
Subgroup analysis revealed that M1a classification (paraaortic lymph node metastasis only) was a consistent weakness across all models, with sensitivity and PPV significantly lower than for M1b and M1c. This reflects both the small number of M1a cases in the simulated cohort (only 12 patients) and the inherent complexity of distinguishing isolated paraaortic nodes from other metastatic patterns.
Ensemble voting aggregates the majority answer from three independent query rounds to produce a final prediction. The study found that ensemble voting achieved non-inferior accuracy compared to any single query round in 44 to 90 percent of comparisons depending on the model and task, with higher benefits for IR tasks than RA tasks.
The benefit of ensemble voting is particularly important because LLMs can produce different answers to identical inputs due to their inherent probabilistic nature. A single query might yield an incorrect answer by chance, but if two of three queries give the same correct answer, the ensemble voting correctly overrides the outlier.
Importantly, the study found that consistency (ICC) and accuracy tracked together across models. Models that were more accurate were also more consistent across repeated queries, suggesting that for LLMs in clinical applications, reliability and validity are closely linked qualities rather than independent properties.
In practical terms, implementing a three-round query with ensemble voting adds minimal cost (approximately three times the API cost per case) while meaningfully improving reliability. The researchers describe a workflow where this entire process can be automated within a clinical decision support system, delivering final ensemble results to physicians without requiring any manual steps.
Despite strong average performance, the study identified specific error patterns that are important for clinical safety. Modality conflicts were a notable error source: when CT or MRI findings contradicted whole-body bone scan results, models sometimes failed to prioritize the higher-specificity cross-sectional imaging over the more sensitive but less specific bone scan, leading to over-staging.
In complex cases with multiple metastatic sites or concurrent secondary malignancies, LLMs occasionally produced conflicting or inconsistent interpretations. For bone metastasis detection specifically, ChatGPT-4-turbo's sensitivity was 96.0% but specificity only 84.8%, meaning roughly 1 in 7 patients without bone metastases would be incorrectly flagged. This false positive rate, while manageable with physician oversight, illustrates why LLMs cannot function independently as diagnostic decision-makers.
A key advantage of general-purpose LLMs compared to custom-trained models is flexibility and low deployment cost. Building a domain-specific model for each clinical task requires large annotated datasets, dedicated AI staff, and significant infrastructure. A single general-purpose LLM API can handle diverse tasks, improves continuously as new model versions are released, and avoids the risk of catastrophic forgetting that can occur when fine-tuning.
The study's main limitation is the use of simulated rather than real patient reports. Report language varies considerably between institutions and individual physicians, and the simulated reports were drawn from limited templates. The findings need validation on real clinical reports from multiple institutions before deployment in clinical settings. The study also used only zero-shot chain-of-thought prompting and did not explore few-shot or retrieval-augmented generation approaches that might further improve performance.
The authors outline a practical clinical decision support system (CDSS) integration model: when a physician activates the system for a patient, it automatically compiles multi-modal reports into predefined prompts, queries the LLM three times via API, calculates ensemble voting results, and delivers final risk classification outputs directly to the physician's workflow with one click.
ChatGPT-4-turbo's performance in the 90 to 92 percent accuracy range across all three RA tasks approaches human expert agreement levels (kappa 0.84 to 0.89), suggesting it could serve as a reliable screening tool to identify non-high-risk patients who might not need aggressive treatment escalation. The model's very high sensitivity and NPV in RA tasks specifically make it well-suited as a rule-out tool.
An important finding is that general-purpose LLMs are rapidly improving across model generations. ChatGPT-3.5-turbo, representing the previous generation, performed at near-chance levels on RA tasks, while ChatGPT-4-turbo reached near-physician accuracy. This trajectory suggests that even higher performance may be achievable with next-generation models without any task-specific training.
The study recommends that implementing LLMs in clinical settings requires clear disclosure of error risks to physicians, strict oversight protocols, and continued validation on real patient cohorts from diverse institutions. The technology is promising enough to warrant piloting in clinical settings, but not yet mature enough to operate without human review of its outputs.