Lung cancer screening saves lives but creates workload. Low-dose computed tomography (LDCT) has become a proven tool for early lung cancer detection, reducing mortality in high-risk populations such as current and former smokers. In 2025 alone, approximately 226,650 new lung cancer cases are projected in the U.S., making efficient screening interpretation a matter of public health urgency.
Lung-RADS standardizes but does not simplify. The American College of Radiology's Lung-RADS reporting system categorizes pulmonary nodules into six groups (1, 2, 3, 4A, 4B, 4X) based on size, density, growth pattern, and morphology. Each category maps to a specific follow-up management pathway. While this standardization is valuable, the actual assignment of categories requires a time-consuming, multistep workflow that introduces inter-reader variability.
Prior AI approaches had critical gaps. Earlier attempts at automating Lung-RADS classification used machine learning models or neural networks that lacked interpretability or addressed only partial classification (e.g., distinguishing suspicious from non-suspicious, or predicting only categories 3 and 4). Studies using early large language models like ChatGPT focused on cancer staging rather than detailed Lung-RADS classification.
The opportunity for large language models. Recent advances in LLMs, particularly their ability to process narrative text and follow complex reasoning guidelines, suggested they could automate Lung-RADS categorization directly from radiology reports. This study investigated whether open-source LLMs with advanced prompt engineering could reliably assign Lung-RADS scores and follow-up recommendations without requiring model fine-tuning.
A large real-world clinical dataset. The study used 10,224 radiology report and impression report pairs from 25,514 patients who underwent LDCT-based lung cancer screening at the University of Florida Health between January 2021 and June 2024. Reports were identified through standardized procedure and diagnosis codes, with pairs retained only when both a radiology report with nodule characteristics and an impression report with a Lung-RADS score were available.
Class imbalance mirrors real-world screening. The distribution of Lung-RADS scores in the dataset reflected real-world prevalence: 77.64% of reports contained Lung-RADS scores of 1 or 2, meaning the vast majority of screened patients had no significant findings. This imbalance posed a challenge for accurately predicting the rarer, higher-risk categories.
Three open-source LLMs evaluated across parameter scales. The study selected models deployable on a single Nvidia A100 GPU: Phi-3.5 (3.8 billion parameters, Microsoft), Phi-4 (14 billion parameters, Microsoft), and Llama 3.3-70B (70 billion parameters, Meta). Model selection prioritized performance on reasoning benchmarks and open accessibility to enable privacy-preserving deployment without sending patient data to external APIs.
Follow-up management categories were consolidated. For management prediction, Lung-RADS categories were grouped by their clinical follow-up action: categories 1 and 2 (12-month LDCT), category 3 (6-month LDCT), category 4A (3-month LDCT), and categories 4B and 4X combined (immediate chest CT). This consolidation aligned predictions with the actual clinical decision being made rather than the underlying category label.
Chain-of-thought prompting as the baseline. Chain-of-thought (COT) prompting introduces intermediate reasoning steps, asking the model to work through the classification logic step by step using instructions derived from lung cancer screening guidelines. This served as the performance baseline for comparison with more complex strategies.
Prompt chaining for structured extraction. Prompt chaining (PC) breaks the task into sequential steps: first extracting nodule characteristics from the radiology report, then using those extracted features to predict the Lung-RADS category. This mirrors the logical workflow a radiologist follows and was hypothesized to improve accuracy by decomposing the problem.
Knowledge-guided variants added guideline context. Two additional strategies (COT-KG and PC-KG) augmented the baseline COT and PC approaches by inserting the actual Lung-RADS guidelines directly into the prompt. This gave the model explicit reference material, enabling in-context learning from the authoritative classification criteria without requiring any fine-tuning on labeled examples.
Three output classification settings were tested. For each prompting strategy, three output formats were evaluated: multiclass classification (assign a single Lung-RADS score), hierarchical classification (first classify benign versus malignant, then predict the specific category), and per-class binary classification (yes/no for each individual Lung-RADS category). The hierarchical approach was motivated by the expectation that splitting the problem into smaller subtasks would improve accuracy.
Llama 3.3-70B led on Lung-RADS classification. For multiclass Lung-RADS score prediction, Llama 3.3-70B combined with either COT-KG or PC-KG achieved the highest micro F1-score of 0.78. Phi-4 followed closely at 0.77, while Phi-3.5 significantly underperformed due to its inability to effectively leverage the in-context information provided in the prompts.
Follow-up management prediction outperformed category classification. For LCS management prediction, Llama 3.3-70B achieved a micro F1-score of 0.86 using COT with guidelines, and 0.87 in hierarchical classification. This consistently higher performance compared to direct Lung-RADS score classification suggests that predicting the clinical action (follow-up interval) is a more tractable task for LLMs than assigning the precise category number.
Hierarchical classification provided minimal benefit. Contrary to the expectation that decomposing benign versus malignant classification first would simplify the overall problem, hierarchical classification produced only marginal improvements for Phi-4 and actually hurt Phi-3.5 performance. The benign-versus-malignant distinction was already well-handled by the models, so the additional decomposition step provided little leverage.
Lung-RADS 4X remained the hardest category. All three models struggled specifically with the Lung-RADS 4X category, which is defined as nodules in categories 3 or 4 with additional suspicious features. The inherent ambiguity of this definition, where neither the published guidelines nor clinicians maintain an exhaustive list of suspicious features, limited the model's ability to learn consistent classification rules.
Speed-performance tradeoff was significant. Llama 3.3-70B required an average of 32.52 seconds per inference in the worst-case scenario (COT with guidelines), compared to 7.93 seconds for Phi-4 and 5.04 seconds for Phi-3.5. Phi-4 emerged as the most practical choice for real-time clinical use, offering near-top performance at roughly one-quarter the inference time of Llama 3.3-70B.
Three main failure categories were identified. An error analysis of 330 misclassified cases from the best-performing setup (Llama 3.3-70B with COT and guidelines) revealed three primary causes: hallucination (32% of errors), failure to follow instructions (23%), and unclear instructions (13%), with remaining errors attributed to miscellaneous causes.
Hallucinations involved plausible but incorrect reasoning. In hallucination cases, the model correctly extracted relevant nodule information and cited the appropriate section of the Lung-RADS guideline, but then produced an incorrect classification. For example, a model might identify that a nodule was new and hazy but then base its prediction solely on size, disregarding the clinically significant characteristics it had accurately identified.
Instruction-following failures had clear examples. In some cases, models ignored explicit rules embedded in the prompt. A notable example involved the interpretation of nodule novelty: when a report lacked a previous size reference, the prompt instructed the model not to classify the nodule as new unless explicitly stated with a previous size of N/A, but models sometimes overrode this rule based on their own inference.
Unclear Lung-RADS 4X criteria created systematic errors. The 4X category proved systematically difficult because neither the published guidelines nor clinical practice provides an exhaustive list of suspicious features that qualify for this designation. This ambiguity propagated directly into model errors when radiological findings like a specific anatomical mass description were not recognized as suspicious features.
In-context learning offers practical deployment advantages. Unlike fine-tuning approaches that require large annotated datasets and periodic retraining, in-context learning can be adapted to different note formats across institutions without model modifications. All models evaluated in this study are open-source, can run offline on institution-owned hardware, and the prompts are publicly available, supporting privacy-preserving deployment without sending patient data to external cloud APIs.
Two collaborative workflow models were proposed. The authors suggested two implementation patterns: an LLM-first workflow where the model provides initial predictions and reasoning that radiologists review and refine, and a radiologist-first workflow where clinicians generate assessments that the LLM validates as a secondary review. Both approaches preserve clinical decision authority while leveraging LLM efficiency gains.
Ground truth quality was an identified limitation. An estimated 12% of cases in the error analysis showed misalignment between the ground truth Lung-RADS scores in the impression notes and what the guideline would indicate, consistent with prior literature reporting at least 6% incorrect Lung-RADS assignments due to subjective nodule characteristic assessments. This introduced uncertainty into the performance evaluation.
Class imbalance and report variation limited performance. The heavy dominance of Lung-RADS 1 and 2 categories (84% combined) created learning challenges for rare high-risk categories. Additionally, inconsistent documentation of nodule size (single dimension versus two-dimensional measurements) and ambiguous prior scan history references introduced systematic interpretation errors that prompt engineering alone could not fully resolve.
First demonstration of LLMs for Lung-RADS classification. To the authors' knowledge, this was the first study to systematically evaluate LLMs for both Lung-RADS categorization and follow-up management classification from narrative radiology reports. The results demonstrate that LLMs with 10 billion or more parameters can achieve clinically meaningful accuracy on these tasks without requiring fine-tuning.
Outperformance over prior automated methods. The LLM-based system outperformed the Vancouver Risk Calculator (92% vs 90% accuracy) and exceeded an earlier artificial neural network approach by 7% accuracy. Unlike black-box neural network models, the LLM-based system also provides explanations for each prediction, supporting radiologist review and building clinical trust in the automated output.
Future work toward smaller, fine-tuned models. The study identified fine-tuning as the next logical step for adapting smaller models like Phi-3.5 to the Lung-RADS task, potentially delivering faster inference with improved accuracy. Addressing class imbalance for rare high-risk categories and incorporating previous radiology reports for temporal context were also identified as key improvements for future system iterations.