Core Innovation This study from Memorial Sloan Kettering Cancer Center demonstrates that high-performing lung cancer detection and segmentation models can be built entirely from mined clinical imaging annotations - without requiring radiologists to manually label a single training image.
The MODS Framework Mining, Object Detection, and Segmentation (MODS) extracts existing line measurement annotations that radiologists make during routine clinical care, converts them to bounding boxes, and uses them as training labels for an object detection model, eliminating the annotation bottleneck.
SAM and MedSAM Integration The Segment Anything Model (SAM) - trained on 1.1 billion natural image masks - and its medical imaging variant MedSAM are integrated into MODS as zero-shot segmentation models, providing tumor contours without tumor-specific segmentation training data.
Scale of the Dataset 10,000 chest CT scans from lung cancer patients at MSK were mined, yielding 10,789 line annotations converted to 5,403 training bounding boxes - a scalable approach that leverages the enormous archive of routine clinical imaging data.
What PACS Contains Picture Archiving and Communication Systems store not just CT images but also radiologist annotations - including the line measurements (longest tumor diameter) that radiologists routinely make when reporting CT scans. These annotations were extracted as the training signal.
Bounding Box Conversion Each line measurement was geometrically converted to a square bounding box. Boxes smaller than 1 cm (too small for reliable 2D detection) or larger than 7 cm (likely non-tumor annotations like biopsy needle markings) were excluded from training.
Quality Assessment Manual review of 500 randomly selected mined annotations confirmed that over 80% annotated pulmonary nodules or masses, with the remainder distributed between pleural masses and lymph nodes. The false positive rate from non-tumor annotations was under 5%.
Training Set Construction To mitigate class imbalance, annotated images were augmented with unannotated images from the same CT scans. Patients were split at the subject level (all scans from one patient in either train or test) to prevent data leakage.
The Incomplete Labeling Problem Radiologists typically annotate only the index lesion (one or a few representative tumor slices), not every visible tumor across all CT slices. This leaves most tumor-containing image slices unlabeled in the training set.
Teacher Model A baseline 'teacher' YOLO v8 model was first trained on the mined annotations, achieving an F1 score of 0.847 on the internal test set. This model was then applied to all unlabeled image slices from the training CTs.
Pseudo-Label Generation The teacher model generated bounding box predictions (pseudo-labels) for 637,170 previously unlabeled slices, adding 66,993 new annotations to the training set. Manual review confirmed that over 70% of these pseudo-labels correctly annotated pulmonary nodules or masses.
Student Model The final 'student' model trained on the combined original and pseudo-labeled dataset improved the F1 score to 0.860, confirming that self-labeling effectively expanded the training set with high-quality annotations.
SAM Architecture SAM combines an image encoder that produces a feature map and a prompt encoder that converts bounding boxes (from the detection model) into embeddings. These are fused to predict a segmentation mask without any tumor-specific training.
Detection-Prompted Segmentation The bounding boxes generated by the student detection model served as prompts to SAM and MedSAM, directing the segmentation model to the tumor region and producing pixel-level tumor contours.
Internal Test Performance On the internal test set, SAM achieved a Dice similarity coefficient (DSC) of 0.842 and MedSAM achieved 0.822 - both indicating high segmentation accuracy for a model that required no tumor segmentation training data.
External Validation On the external LIDC-IDRI dataset (979 CT scans from 7 medical centers), the fine-tuned student model paired with SAM/MedSAM achieved F1 of 0.832 and DSCs of 0.802/0.804 - performance comparable to fully supervised models trained on LIDC-IDRI ground truth masks.
Surveillance Monitoring Automated detection and segmentation models can track the growth kinetics of pulmonary nodules across serial CT scans, flagging lesions that have grown beyond threshold sizes without requiring radiologist review of every scan.
Radiomic Analysis Accurate automated segmentation enables radiomic feature extraction without manual contouring - a key bottleneck in clinical radiomic workflows. MODS-generated contours could seed large-scale radiomic studies from existing clinical CT archives.
Treatment Response Assessment In patients receiving systemic therapy, automated tumor segmentation at each follow-up CT enables objective response assessment by RECIST criteria (measuring longest tumor diameter) and volumetric response tracking.
Scalability Advantage The MODS approach scales to any institution with a PACS system, requiring only the extraction of existing line measurement annotations - no new annotation effort is needed. This makes it practical for widespread adoption across diverse healthcare settings.
2D Slice-Based Model The current YOLO-based detection operates on single 2D CT slices, potentially missing small nodules visible only when integrating information across multiple adjacent slices. A 3D detection architecture would improve sensitivity for sub-centimeter lesions.
Domain Shift Challenge Performance decreased on the external LIDC-IDRI dataset before fine-tuning, reflecting the impact of different CT protocols and patient populations. Developing domain-adaptation techniques that don't require any external labeling would extend MODS applicability.
Nodule vs. Mass Distinction The detection model does not distinguish between nodules, masses, and other pulmonary abnormalities beyond size thresholds. Future work should classify detected findings by morphology to provide clinically actionable characterization.
Extending MODS to Other Body Sites The MODS framework has already been demonstrated for brain tumors in prior work. Future extensions to liver, kidney, and pancreatic lesions - where radiology measurements are also routinely made - could create a universal clinical data mining platform.