Lung cancer is the leading cause of cancer death worldwide, and prognosis guides treatment. After surgical resection, determining which patients are at high enough risk of recurrence to benefit from adjuvant chemotherapy or radiotherapy is a critical and difficult clinical decision. The standard TNM staging system provides population-level guidance but has been described as too broad for individualized treatment planning.
Deep learning models can improve risk stratification beyond staging alone. Studies have shown that clinical features, PET/CT imaging data, and pathological variables each contribute independently to predicting lung cancer outcomes. Combining these data streams in a deep learning model (DLM) offers more precise and personalized prognostication than any single data source alone.
The challenge is integrating these models into clinical workflows. Despite promising research, few studies have developed actual clinical interfaces for lung cancer prognostic models, and none had previously incorporated multi-modality imaging alongside surgical and pathological information into a system evaluated by real clinicians. This gap between model development and clinical adoption motivated the current study.
The CDSS integrates PET/CT imaging with clinical, surgical, and pathological information. The system displays each patient's preoperative FDG-PET/CT scan in axial, sagittal, and coronal planes, alongside their clinical variables (age, sex, smoking history), surgical details (resection type, margin status), and pathological tumor characteristics (histology, stage, lymphovascular invasion, visceral pleural invasion).
The interface was built in MATLAB with Python integration for the deep learning model. A single button labeled Show Model Risk loads the patient's data into the validated DLM, generating a prognosis score and a risk category (high, medium, or low) within approximately ten seconds. The interface color-codes the result: red for high-risk, orange for medium-risk, and green for low-risk patients, making the output immediately interpretable.
Kaplan-Meier survival curves for each risk group are displayed alongside the prediction. These curves, derived from external validation data of 195 patients, show expected recurrence-free survival and overall survival for patients in the predicted risk category, providing clinicians with population-level context for the individual patient's predicted prognosis.
The study cohort included 10 NSCLC patients from a single center. Patients had pathologically confirmed stage I to III non-small cell lung cancer and had undergone upfront surgical resection between 2012 and 2018. They were randomly selected from the testing dataset of the previously validated DLM to ensure an unbiased evaluation of the CDSS.
Four oncologists assessed all ten patients twice: once without and once with the DLM output. The study included two thoracic surgical oncologists and two medical oncologists with experience ranging from 4 to 30 years. Each clinician first reviewed all patients using only the imaging and clinical information, then reviewed the same patients in a new randomized order with the DLM risk prediction displayed.
Clinicians recorded risk scores, confidence levels, and treatment recommendations for each patient. For each case, users assigned a numeric probability of treatment failure (0 to 100), a categorical risk group (high, intermediate, or low), a confidence level (from none to completely), and a post-surgical recommendation (surveillance, adjuvant therapy, or multidisciplinary team review). Responses were locked after submission.
Statistical analysis compared performance before and after CDSS use. The area under the receiver operating curve (AUC) was used to measure prognostic accuracy for each clinician. The DeLong test assessed whether individual clinician AUCs differed significantly from chance and from their own performance without the DLM. Inter-user agreement was measured using the Fleiss-Kappa statistic.
Qualitative interviews explored clinician attitudes toward AI in clinical practice. After completing the study, all four clinicians participated in semi-structured audio-recorded interviews covering topics including CDSS usability, trust in prognostic models, the role of explainable AI, barriers to clinical implementation, and the use of prognostic scores in patient communication.
The DLM alone achieved an AUC of 0.83 for recurrence prediction. Without any clinician input, the deep learning model was a statistically significant predictor of treatment failure in the 10-patient cohort (AUC 0.83, 95% CI 0.53-1, p = 0.03). This establishes the standalone performance of the model as a baseline for comparison with clinician performance.
Clinician performance improved numerically after using the CDSS, but the change was not statistically significant. All four users showed positive delta-AUC values when comparing their risk group stratification before and after using the DLM information, meaning all four performed at least as well or better with the AI assistance. However, only one improvement reached statistical significance (p = 0.01 for one user's risk group AUC).
Three out of four clinicians achieved higher AUC than the DLM alone when using the CDSS. When clinicians integrated the AI prediction with their own clinical judgment rather than relying on the model alone, their combined performance exceeded the standalone DLM for most users. This suggests that human-AI collaboration, rather than AI replacement of clinical judgment, is the optimal use case.
Medical oncologists showed a greater improvement trend than surgical oncologists. The two medical oncologists showed numerically larger improvements in stratification performance after using the CDSS DLM information (p = 0.07, approaching significance). This may reflect a difference in how each specialty incorporates prognostic models relative to their clinical experience and decision frameworks.
Inter-clinician agreement decreased after DLM use, reflecting individualized interpretation of AI output. Without the CDSS, risk group predictions showed moderate agreement among clinicians (Fleiss-Kappa 0.42). After using the DLM, agreement decreased to slight (kappa 0.18), as clinicians weighted the AI output differently based on their clinical judgment and specialty background.
A key case illustrates how DLM information can reveal non-obvious high-risk patients. One stage IB patient with no conventional high-risk post-surgical features was classified by most clinicians as intermediate risk without the CDSS. After viewing the DLM output, three of four clinicians reclassified this patient as high-risk. This patient did in fact experience recurrence, demonstrating that the model identified prognostic signal missed by conventional clinical assessment.
Clinician confidence increased numerically with DLM use but without statistical significance. The proportion of clinicians selecting very confident increased after using the CDSS, while the proportion choosing moderately confident decreased. This trend did not reach statistical significance (p = 0.22), likely due to the small sample size.
Medical oncologists significantly increased agreement on treatment recommendations when using the DLM. For the two medical oncologists, recommendation agreement rose from moderate (kappa 0.49) to substantial (kappa 0.73) with DLM information. Surgical oncologists showed the opposite trend, with their agreement decreasing, suggesting that the DLM information is integrated differently across specialties and may prompt further individualized clinical reasoning rather than consensus behavior.
All four clinicians rated the CDSS interface highly for usability and time efficiency. Average ratings were eight out of ten for user-friendliness and nine out of ten for time efficiency. Clinicians particularly valued the integration of imaging, pathology reports, and clinical data in a single interface, the ability to inspect PET SUV values on a voxel-by-voxel basis, and the color-coded risk presentation.
Clinicians trusted the model when it identified unexpected high-risk patients. One clinician noted that a high-risk DLM prediction that contradicted their initial impression made them more likely to refer the patient for multidisciplinary team review, an outcome that would not have occurred otherwise. However, the same clinician was more skeptical when the model predicted low risk contrary to their expectation, indicating that human judgment remains a strong filter on model output.
Validation and local data performance were identified as the primary barriers to clinical adoption. All four clinicians expressed willingness to use validated prognostic models in practice but cited the absence of rigorous prospective validation and local data validation as key concerns. Differences in imaging acquisition protocols and data quality across institutions mean that external validation alone is insufficient for confident local deployment.
Attitudes toward explainable AI were mixed and cautiously skeptical. When asked whether seeing the model's internal reasoning would improve their trust, most clinicians expressed hesitation. They noted that saliency maps and other explainability outputs would need their own validation and could be confusing or misleading without appropriate training in their interpretation.
This is the first study to integrate a multi-modality DLM for NSCLC prognostication into a clinical interface and evaluate it with real clinicians. Previous CDSS studies for lung cancer used only clinical variables without imaging, or validated underlying models without building a usable frontend interface. This study bridges that gap by demonstrating that a multi-modality system is both technically feasible and well-received by oncologists.
Human-AI collaboration outperformed both human-only and AI-only approaches. The finding that most clinicians exceeded the DLM's standalone AUC when using CDSS assistance is consistent with broader evidence that AI tools work best as decision-support systems rather than autonomous decision-makers. The AI highlights patterns that clinicians can then interpret through their clinical knowledge of the individual patient.
Decreased inter-clinician agreement after DLM use may reflect appropriate individualization rather than inconsistency. When a model produces an unexpected result, different clinicians will weigh it differently based on their specialty, experience, and knowledge of the patient's full clinical picture. This variability may actually represent optimal use of the tool, with each clinician updating their prior beliefs by different amounts based on the model output.
Future systems should incorporate genomic markers and integrate with hospital PACS. The study was conducted retrospectively before the routine use of neoadjuvant immunotherapy and genomic testing (EGFR, ALK alterations), which are now central to NSCLC management. Future CDSS versions must include these variables and integrate directly with clinical picture archiving systems to minimize workflow friction and ensure the completeness of input data.
The CDSS showed promising but non-significant improvements in lung cancer prognostication. Across all four clinicians, the AI-assisted CDSS produced numerical improvements in recurrence risk stratification and prediction confidence. The limited patient cohort of 10 cases and small user group of 4 clinicians were insufficient to achieve statistical significance, but the consistent directional improvement is encouraging.
Clinician feedback will directly improve future CDSS design. Specific recommendations from the study included eliminating the raw prognosis score in favor of risk groups and recurrence rates, incorporating patient performance status metrics, providing validation statistics within the interface, and integrating the system with clinical PACS for seamless data exchange.
The study establishes a foundation for future multi-modality CDSS clinical trials. By demonstrating feasibility, identifying design improvements, and documenting clinician attitudes toward AI-assisted prognostication in the Canadian clinical context, this work provides a roadmap for the larger prospective trials needed to definitively establish the clinical value of AI decision-support in resected NSCLC management.