A major data problem. Bladder cancer is the third most prevalent non-cutaneous cancer in the United States, and most patients live for more than nine years after diagnosis. During this time, they undergo regular tumor surveillance, but current guideline recommendations are based on limited, older data from highly selected patient populations in European trials from the 1980s and 1990s.
Existing population-based databases like SEER and SEER-Medicare only capture pathology information at the time of diagnosis. This severely limits the ability to study how surveillance care influences recurrence and progression over time -- exactly the outcomes most relevant to patients and clinicians.
To truly understand how surveillance affects outcomes, researchers need population-based cohorts with longitudinal pathology data collected at every biopsy visit. Assembling such cohorts from tens of thousands of free-text pathology reports requires automation that was not previously available.
Automating report abstraction. The Department of Veterans Affairs maintains a national Corporate Data Warehouse containing full-text pathology reports for patients across its entire healthcare network. This represents an exceptional opportunity to assemble large, longitudinal cohorts -- but only if key pathologic details can be automatically extracted from thousands of reports.
Natural language processing (NLP) is a field of computer science that enables machines to understand and extract structured information from unstructured human-written text. The researchers developed an NLP engine specifically designed to read bladder pathology reports and pull out clinically critical variables including histology type, tumor grade, invasion status, invasion depth, presence of carcinoma in situ, and whether muscularis propria was present in the specimen.
The study proceeded in three stages: developing a gold standard through expert human annotation, building and refining the NLP engine, and finally applying the validated engine to a large national cohort of over 10,000 bladder cancer patients.
Starting with human expertise. Six hundred bladder pathology reports were randomly selected from VA patients diagnosed with bladder cancer between 2005 and 2011. These reports were independently reviewed by two expert annotators -- a urologic oncologist and a urology chief resident -- using a specialized annotation tool called ChartReview.
Each annotator highlighted and categorized statements in every report covering six key variables: histology type, invasion presence or absence, invasion depth, tumor grade, carcinoma in situ presence, and whether muscularis propria was present in the specimen. When the two annotators disagreed, a third independent urologic oncologist served as arbiter to determine the final categorization.
Inter-rater reliability between the two annotators was excellent across all variables, with Cohen's kappa values ranging from 0.82 to 0.90. This annotated dataset of 600 reports served as the gold standard against which all NLP performance was measured.
The 600 reports were then split into a training set of 300 reports for rule design, a development set of 150 reports for refining and testing rules, and a final validation set of 150 reports for the official performance evaluation.
Rule-based architecture. The NLP engine was built within the VA Corporate Data Warehouse using Apache UIMA AS and the Leo framework -- enterprise-level tools for processing large volumes of unstructured text. Rather than using machine learning, the team built a rule-based system where the rules were derived directly from actual language used by pathologists in the training reports.
The pipeline used regular expressions to find specific phrases associated with each clinical concept. Once a relevant phrase was identified, additional heuristics analyzed the surrounding context to determine if the mention was negated (for example, 'no invasion was found'), historical ('history of urothelial carcinoma'), or otherwise irrelevant to the current specimen.
Invasion was particularly complex to handle because negation could mean different things: 'no lamina propria or muscularis propria invasion' was interpreted as complete absence of invasion, while 'no muscularis propria invasion' was interpreted as the tumor being limited to lamina propria. Separate pipelines were developed for each variable, with invasion presence and depth handled together to manage these distinctions.
After applying the validated engine to 31,009 non-annotated reports, a post-hoc cleaning step was performed to remove specific patterns that caused errors, such as anatomical directional terms ('superior to inferior') being misclassified as references to superficial invasion.
High accuracy across most variables. On the 150-report validation set, the NLP engine achieved accuracy of 0.96 for histology, 0.93 for grade, 0.98 for carcinoma in situ, and 0.87 for invasion presence versus absence. For these four variables, both positive predictive value and sensitivity were mostly above 0.85, meaning the engine correctly identified cases when it made a positive call and captured the vast majority of actual positive cases.
The most challenging variable was invasion depth, where accuracy fell to 0.68. Sensitivity was particularly low for muscularis propria invasion at only 0.57, meaning the engine missed about 43% of truly muscle-invasive cases. This was attributed to the high linguistic variability pathologists use to describe invasion depth -- some variables had 261 to 863 unique expressions per 1,000 reports compared to only 30 to 40 for simpler variables like histology type.
Despite the challenges with invasion depth, positive predictive value remained acceptable at 0.82 for lamina propria invasion and 0.87 for muscularis propria invasion. This means that when the engine called a case muscle-invasive, it was correct 87% of the time -- an important property for population-level research.
Near-complete coverage. When the validated engine was applied to all 31,009 non-annotated bladder pathology reports for 10,725 VA bladder cancer patients, it successfully retrieved information from 98% of reports and 99% of patients. Only about 511 reports out of more than 31,000 contained no extractable information.
In a random sample of 100 reports each for four key classifications, the positive predictive values in the full dataset were 0.99 for no cancer in the biopsy, 0.96 for non-invasive disease, 0.95 for invasive disease, and 0.91 for lamina propria invasion. These high precision values mean researchers can assemble cohorts with confidence that included patients actually have the pathology they are believed to have.
The distribution of pathologic findings in the full 30,498-report dataset mirrored the annotated 600-report sample closely, suggesting the engine was applying consistent logic at scale. Among urothelial carcinoma reports, 38% were non-invasive, 44% invasive, and 13% showed carcinoma in situ -- distributions consistent with the known epidemiology of bladder cancer.
The core technical problem. The biggest obstacle to accurate extraction was the extraordinary variability in how pathologists describe invasion depth. The engine encountered 261 unique expressions per 1,000 reports for muscularis propria invasion and 863 unique expressions per 1,000 reports for perivesical invasion -- compared to only 30 unique expressions for urothelial histology and 32 for carcinoma in situ.
This variability is partly because most VA bladder pathology reports do not use standardized synoptic formats recommended by the College of American Pathologists. When pathologists write free text rather than filling in standardized fields, the range of phrasing becomes enormous and difficult for rule-based systems to capture completely.
The authors note that wider adoption of standardized pathology reporting would benefit both NLP accuracy and clinical care. In the meantime, the acceptable positive predictive values achieved -- even for the most challenging variables -- demonstrate that population-level research is feasible with the current engine.
A platform for future studies. The validated NLP engine enables researchers to assemble population-based cohorts of bladder cancer patients with longitudinal pathology data for the first time. Rather than relying on single-point diagnosis data, studies can now track how pathologic findings evolve across repeated biopsies over years of surveillance.
Specific research applications include identifying cohorts of patients with non-invasive low-grade versus high-grade disease and studying how different surveillance patterns -- frequency of cystoscopy, timing of biopsies -- affect time to recurrence or progression to invasive disease. The excellent sensitivity of 0.97 for invasion presence means the engine can reliably detect when a previously non-invasive patient develops invasive disease.
The authors also note potential clinical applications such as incorporation into automated risk prediction tools, though they emphasize that different use cases may require different NLP configurations. External validation outside the VA system would be needed before deploying the engine in other healthcare settings, as report structure and vocabulary may differ across institutions.
This work represents a foundational infrastructure step: the engine itself is not the final output, but rather the tool that makes large-scale, longitudinal bladder cancer health services research possible at a national population level.
A validated tool ready for deployment. The study successfully developed and validated an NLP engine capable of abstracting six clinically important pathologic variables from full-text bladder cancer pathology reports with high accuracy for five of the six variables. The engine processed data for 99% of patients in a large national cohort.
The limitation of lower accuracy for invasion depth is understood and quantified, allowing future researchers to account for this in their study designs. For most population-level research questions -- particularly those focused on non-muscle invasive disease -- the engine's high precision values are sufficient to support valid scientific conclusions.
The immediate next step is using the abstracted data to study how different patterns of surveillance care, including frequency and timing of cystoscopy and biopsy, impact time to recurrence and progression to invasive disease. This represents a major advance in bladder cancer health services research that was not previously possible using available data sources.