AJCC (American Joint Committee on Cancer) staging is the international standard for classifying cancer severity and guiding treatment decisions. For prostate cancer, staging requires integrating the TNM classification (tumor extent, lymph node involvement, and metastasis), Gleason grade groups, and PSA levels according to the 8th edition AJCC manual.
After radical prostatectomy (surgical removal of the prostate), the pathology report on the removed specimen becomes a critical document determining prognosis and whether adjuvant therapy is needed. These reports contain measurements of extraprostatic extension (EPE), seminal vesicle invasion, lymph node counts, resection margins, and Gleason patterns, all required for accurate staging.
The problem is that most pathology reports are written in free-text format, meaning the relevant staging variables are embedded in narrative prose rather than structured fields. Extracting these elements manually for every case is time-consuming and introduces variability between reviewers. Building specialized natural language processing models for this task has historically required large annotated training datasets that are expensive to create.
Large language models (LLMs) offer an alternative: pre-trained on vast text corpora, they can extract structured information from narrative reports using simple prompt instructions, without requiring task-specific training. Combined with a knowledge graph that encodes medical rules, this approach could automate staging while simultaneously validating logical consistency in the extracted data.
The system was tested on 152 radical prostatectomy pathology reports from a Korean hospital (internal dataset) and 88 reports from The Cancer Genome Atlas (TCGA) representing diverse international institutions (external validation). Information extraction used Mistral-Small-3.2-24B-Instruct, an open-source LLM running locally on dual GPU hardware, to extract 16 clinical parameters from each report using zero-shot prompting.
Zero-shot prompting means the model receives instructions in plain language specifying which parameters to extract and what format to return them in, without any example cases showing correct extractions. The prompt frames the model as a senior genitourinary pathologist and requests output in structured JSON format with each extracted value, a supporting evidence quote from the report text, and a confidence score from 0 to 1. This privacy-preserving approach keeps patient data on local infrastructure rather than sending it to cloud APIs.
The knowledge graph was built using Neo4j, a graph database system. It represents each patient, pathology report, TNM staging elements, Gleason grade, PSA level, and final AJCC stage as distinct nodes connected by typed relationships such as HAS_T_STAGE, HAS_GRADE, and CLASSIFIED_AS. Once LLM-extracted data populate these nodes, a hierarchical rule-based algorithm following AJCC 8th edition criteria assigns the final stage automatically based on the combination of TNM, grade, and PSA values.
The knowledge graph also implements consistency validation rules that check the extracted data for logical contradictions. For example, a T2 stage with documented extraprostatic extension is impossible because EPE automatically elevates a tumor to at least T3a. These rules generate specific error codes when contradictions are found, creating an auditable record of data quality problems that a clinician can review and correct.
Applied to 150 internal dataset reports (2,400 field-document pairs), the Mistral model achieved overall micro-averaged accuracy of 0.973 and F1-score of 0.986. Six individual parameters achieved perfect or near-perfect scores: M-stage and seminal vesicle invasion both reached F1-score of 1.000, while lymph node metastasis count, N-stage, tertiary Gleason pattern, and perineural invasion all exceeded 0.997.
The weakest performance was on Percentage of Secondary Gleason Pattern, which achieved accuracy of 0.907 and F1-score of 0.951. A typical error occurred when the report stated a Gleason score of 4+3 with 60% pattern 4: the model correctly extracted 60% for the primary pattern but failed to infer that the secondary pattern represented the remaining 40%. This requires a calculation step beyond direct text extraction, illustrating a conceptual limit of the zero-shot approach.
In external validation on 88 TCGA reports from diverse institutions worldwide, overall performance remained strong with accuracy of 0.938 and F1-score of 0.968. Twelve of the 16 parameters achieved F1-scores above 0.94 on the external dataset, demonstrating that the approach generalizes well beyond the institution where the system was developed.
The largest drop in external validation was in M-stage extraction (F1-score 0.819), primarily because TCGA reports frequently record M-stage using notation like pMx, which the LLM sometimes treated as not mentioned rather than as an explicit metastasis assessment. This reveals a systematic difference in reporting conventions between institutions that affects extraction accuracy for specific parameters.
Using LLM-extracted data as input to the knowledge graph's rule-based staging algorithm, the system achieved macro-averaged F1-score of 0.930 for AJCC staging on the internal dataset. Stage IIIC achieved perfect F1-score of 1.000, and Stage IVA reached 0.960, while common stages like IIIB (0.950) and IIB (0.957) also performed strongly.
The weakest staging performance was for cases labeled Unknown (F1-score 0.815), where the model could not assign a definitive stage due to missing required parameters. On external validation, overall macro-averaged staging F1-score dropped to 0.833, and Unknown cases fell further to 0.455, primarily because M-stage labeling conventions differed between datasets, causing the LLM to mark pMx values as not mentioned, which the system then treated as unclassifiable.
Knowledge graph consistency validation identified 5 cases (3.3%) with logical staging errors among the 150 internal reports processed. The most common error was T3a staging documented without any mention of extraprostatic extension (2 cases), which is a medical impossibility since T3a specifically requires EPE. One case showed a Gleason sum of 7 incorrectly labeled as Grade Group 3 rather than Grade Group 2, because the correct grade was marked with an X symbol in a table that the LLM misinterpreted.
PSA values were available in only 44.7% of internal reports and 29.3% of TCGA reports, reflecting a known documentation gap in pathology practice. PSA is a clinical lab value typically recorded in separate hospital systems, and only makes it into pathology reports when clinicians explicitly include it on the tissue requisition form. Without PSA, certain stage distinctions (such as Stage IIIA, which requires PSA of 20 or above with T1-T2 disease) cannot be made from pathology reports alone.
A defining feature of this system is that it uses a locally deployed open-source LLM rather than a cloud-based commercial API. This means pathology report data never leaves the hospital's infrastructure, addressing a fundamental privacy barrier that has limited the adoption of LLM-based clinical tools. Prior research has shown that open-source models achieve comparable accuracy to closed commercial models like GPT-4 for structured data extraction from pathology reports.
The system has important current limitations. Staging is performed from pathology reports alone, but complete AJCC staging requires imaging data for M-stage assessment: bone scans, CT, or PET-CT are needed to detect distant metastases. Because radical prostatectomy is performed only for localized disease, the study assumed all cases without documented M-stage were M0, which is clinically reasonable but would need to be replaced by actual imaging data in a full clinical deployment.
Only a single open-source LLM was evaluated, and the model encountered specific difficulties with tabulated report formats, nested numeric calculations (like deriving secondary Gleason percentage from the primary percentage), and institution-specific notation variations. Future systems would benefit from benchmarking multiple models, optimizing prompts for specific report templates, and potentially integrating probabilistic reasoning in the knowledge graph for ambiguous findings.
The consistency validation capability is a distinct advantage over simpler extraction systems. When the knowledge graph detects that a T2 stage is inconsistent with documented EPE, it generates a specific error code pointing to exactly which rule was violated. This creates an auditable quality assurance layer that could alert clinicians to review specific cases before staging information is used for treatment decisions, reducing the risk of staging errors propagating through clinical workflows.
The 0.986 F1-score for information extraction and 0.930 for staging classification represent performance levels that approach those of experienced clinicians applying the same standardized criteria. Because the system also produces evidence quotes and confidence scores alongside each extraction, clinicians can quickly audit uncertain fields rather than reviewing the full free-text report for every case.
The combination of LLM extraction and knowledge graph validation creates a two-layer quality control system. The LLM handles the flexibility of natural language understanding, adapting to varying terminology and report styles. The knowledge graph handles the rigidity of medical logic, enforcing rules that must hold regardless of how a report was written. Neither layer alone provides both capabilities.
This architecture is explicitly designed for extensibility to other cancer types. The knowledge graph schema can be updated with different staging rules, and the LLM prompt can be modified to extract different parameters, allowing the same pipeline to serve bladder cancer, kidney cancer, colorectal cancer, or any other malignancy with a defined staging system. This makes the development investment reusable across an oncology department's full case volume.
For healthcare systems with large case volumes, automated staging could reduce the manual work of clinicians who currently extract and enter pathology parameters into staging databases, freeing time for tasks requiring clinical judgment. The 3.3% inconsistency rate detected by the knowledge graph also suggests that manual staging workflows currently contain comparable rates of logical errors that go undetected, providing an additional quality assurance benefit beyond automation efficiency.