Pancreatic ductal adenocarcinoma (PDAC) is typically diagnosed at an advanced stage because early-stage disease produces few specific symptoms and no validated biomarkers exist for general population screening. The five-year survival rate for late-stage disease remains below 5%, compared to over 30% if diagnosed at a localized stage.
Electronic health records (EHRs) generated through routine clinical care contain longitudinal data on symptoms, laboratory values, prescriptions, and diagnoses that collectively may signal early pancreatic cancer development months or years before a formal diagnosis. Analyzing these data with predictive models could enable earlier detection.
Multiple modeling approaches have been proposed for EHR-based cancer prediction. Traditional Cox proportional hazards regression is statistically interpretable and well-understood, but may not capture complex non-linear relationships in the data. Machine learning methods such as Random Survival Forests (RSF) and eXtreme Gradient Boosting (XGB) can model non-linearities but may sacrifice interpretability and calibration.
This study directly compared these three modeling approaches across two large independent cohorts to understand their relative strengths and whether machine learning truly adds value over standard regression for PDAC prediction from EHR data.
The study utilized two independent retrospective cohorts. The first was from Kaiser Permanente Southern California (KPSC), comprising approximately 1.8 million patients with 1,792 incident PDAC cases. The second was from the Veterans Affairs (VA) health system, comprising approximately 2.7 million patients with 4,582 incident PDAC cases.
Using both cohorts independently allowed assessment of whether predictive models built in one health system generalize to another with very different patient demographics: KPSC serves a predominantly commercial insurance population in Southern California, while VA serves predominantly male military veterans with different comorbidity profiles.
Features extracted from EHR data included age, abdominal pain episodes, weight loss, HbA1c levels (a diabetes marker), ALT levels (a liver function marker), and other clinical variables that prior research had suggested might herald PDAC in the preceding year or two before diagnosis.
Models were trained on cases diagnosed within a defined window and evaluated using time-dependent AUC and calibration metrics. Calibration measures whether the model's predicted probabilities match observed event rates, a critical requirement for clinical use beyond simple ranking.
In the KPSC cohort, XGB achieved the highest AUC of 0.779, followed by RSF at 0.767, and Cox regression at 0.737. The differences between XGB and RSF were modest, but both substantially outperformed Cox regression in discrimination, confirming that machine learning captures predictive signal not fully accessible to linear models.
However, despite its superior discrimination, XGB was poorly calibrated: its predicted probabilities did not accurately reflect observed PDAC rates. This means that while XGB was better at ranking patients by relative risk, its absolute probability estimates could not be directly used for clinical risk counseling or setting intervention thresholds.
Cox regression showed better calibration despite lower discrimination. This trade-off is important for clinical applications: absolute risk estimates (e.g., a patient has a 3% 2-year risk) are more directly actionable for surveillance enrollment decisions than rankings alone.
Results were broadly consistent across both KPSC and VA cohorts, with similar relative performance ordering. This cross-cohort consistency strengthens confidence that the performance differences reflect genuine model properties rather than dataset-specific artifacts.
Across all three modeling approaches, a consistent set of clinically meaningful predictors emerged as most important for PDAC risk: increasing age, new-onset or worsening abdominal pain, unexplained weight loss or weight change, elevated HbA1c (reflecting new-onset diabetes), and elevated ALT (suggesting biliary or hepatic involvement).
The finding that new-onset diabetes (reflected by HbA1c elevation) is a significant predictor aligns with substantial prior evidence linking new-onset diabetes in older adults with early PDAC. Pancreatic tumors may disrupt islet cell function before becoming clinically apparent, making new-onset diabetes a potential early warning sign.
Unexplained weight loss is a well-recognized general cancer warning sign, but its quantification in EHR data as a predictive feature demonstrates that routine vital sign documentation has underutilized predictive value for identifying patients at risk of serious underlying malignancy.
The convergence of these predictors across RSF, XGB, and Cox regression provides strong cross-validation of their importance: when multiple diverse modeling approaches agree on which variables matter, the robustness of those associations is substantially more credible than if they emerged from a single method.
The study's findings suggest that machine learning and regression models are not substitutes but complements in EHR-based cancer prediction. XGB's superior discrimination makes it better for identifying the highest-risk individuals from a large population, while Cox regression's calibration makes it better for communicating absolute risk to individual patients.
A practical deployment strategy could use XGB for population-level screening to generate a ranked list of high-risk patients for outreach, while Cox regression is used to communicate individualized risk estimates to patients and clinicians once a high-risk individual is identified.
The poor calibration of XGB is a known limitation of gradient boosting methods in survival analysis settings with rare events and requires careful handling if absolute risk estimates are needed. Calibration post-processing techniques such as Platt scaling or isotonic regression can partially address this issue.
More broadly, this study is a useful reminder that no single performance metric is sufficient for evaluating clinical prediction models. Discrimination (AUC) and calibration are both necessary, and their relative importance depends on the specific clinical use case the model is intended to support.
This large two-cohort comparison demonstrated that machine learning approaches modestly but meaningfully outperform Cox regression for PDAC risk discrimination from EHR data, with XGB achieving AUC 0.779 and RSF 0.767 versus Cox at 0.737. However, calibration differences complicate the choice of method for clinical deployment.
The consistent predictors identified across both models and both cohorts provide a foundation for developing a validated clinical risk score for PDAC that could be automatically computed from routine EHR data and used to trigger enhanced screening conversations or referrals.
Future work should focus on prospective validation and clinical utility studies that test whether acting on model-identified high-risk patients (through enhanced surveillance or diagnostic workup) actually improves stage at diagnosis and survival outcomes.
Given the scale of the cohorts analyzed (over 4 million patients combined) and the consistency of findings across two very different health systems, this study provides one of the most rigorous evaluations of EHR-based PDAC prediction modeling to date, offering a strong evidence base for next-stage clinical implementation research.