The clinical need for precise segmentation The liver is the largest solid organ in the body and is involved in both primary cancers (hepatocellular carcinoma, cholangiocarcinoma) and metastases from colorectal, breast, and lung cancer. Standard response criteria (RECIST) require measuring the diameter of the largest target lesion, making accurate, reproducible segmentation of focal lesions essential for cancer diagnosis, treatment planning, and monitoring.
Why manual segmentation is insufficient Manual delineation of liver tumors in 3D CT scans is time-consuming, poorly reproducible, and highly operator-dependent. Localizing tumor lesions is a prerequisite for treatment options including thermal ablation, radiotherapy, surgical resection, and arterial embolization. These clinical requirements create a strong demand for reliable automated segmentation.
The LiTS benchmark was created to fill this gap The Liver Tumor Segmentation Benchmark (LiTS) was organized in three events - at ISBI 2017, MICCAI 2017, and the Medical Segmentation Decathlon at MICCAI 2018 - to systematically evaluate the state of the art in automated liver and liver tumor segmentation from CT images. 75 algorithm submissions were received across the three events.
Key results at a glance The best liver segmentation algorithm achieved a Dice score of 0.963 - near expert-level. Liver tumor segmentation proved far harder: best Dice scores were 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018), with lesion detection recall reaching only 0.554 at best. No single algorithm performed best for both liver and tumor segmentation across all three events.
Multi-center dataset design The LiTS dataset was collected from seven clinical sites spanning Germany, the Netherlands, Canada, Israel, and France. It contains 201 abdominal CT volumes, of which 194 contain lesions. The training-to-test split was approximately 2:1 (131 training, 70 test), with similar institutional representation in both sets.
Diverse tumor types and imaging conditions The dataset includes primary liver tumors (HCC, cholangiocarcinoma) and secondary metastases from colorectal, breast, and lung cancers. Images were acquired with varying contrast agents, CT scanners, and protocols, including pre- and post-therapy scans and those with metal artifacts. In-plane resolution ranges from 0.56 mm to 1.0 mm, and slice thickness from 0.45 mm to 6.0 mm.
Annotation protocol and quality control Each dataset was annotated slice-by-slice by a radiologist with over 3 years of oncologic imaging experience using ITK-SNAP. All lesions regardless of origin were labeled uniformly as 'Tumor'. Three additional readers independently verified annotations, with a senior radiologist serving as tie-breaker. Scans with very small or uncertain lesion-like structures were excluded from annotation.
Online evaluation via CodaLab All submissions were evaluated automatically through the CodaLab platform, which computed Dice scores, average symmetric surface distance (ASD), and relative volume difference (RVD) per case. The platform remained open after the three challenge events and had received over 3,414 valid submissions from more than 900 registered users as of April 2022.
Traditional liver segmentation methods Before deep learning, liver segmentation relied on statistical shape models (SSMs), probabilistic atlases, and graph-cut methods. SSMs captured the general shape of the liver but failed to handle its high variability. Multi-atlas approaches using non-rigid registration and label fusion improved results but remained limited in generalizing to pathological livers.
Traditional tumor segmentation methods Early liver tumor segmentation used thresholding and spatial regularization, active contours, and level sets. Supervised methods included support vector machines, random forests, and AdaBoost classifiers trained on handcrafted texture features. All these methods relied on manually engineered features and were sensitive to imaging variability.
Deep learning changes the field after 2016 Since Christ et al. (2016) introduced the first 3D U-Net for liver and tumor segmentation with a cascaded strategy and conditional random field refinement, deep learning became the dominant approach. U-Net and its variants, with skip connections for preserving spatial detail, were used in nearly all top-performing LiTS submissions. In 2017, 2D and 2.5D inputs dominated due to computational constraints, but by 2018 3D convolutional neural networks became feasible and outperformed shallower approaches.
Common design choices of top performers The highest-performing algorithms shared several features: cascaded coarse-to-fine pipelines using one network for liver detection and a second for tumor segmentation, data augmentation with geometric transformations, HU-value clipping and normalization, and post-processing to remove false positives outside the liver mask. Ensemble methods combining multiple models also improved results, particularly in MICCAI 2017.
Liver segmentation is near-solved Nearly all participating teams achieved Dice scores above 0.920 for liver segmentation, with the best method reaching 0.963. The margin between top teams was small - differences in ranking appeared more prominently with the ASD metric than with Dice. Methods specifically optimized for liver (MICCAI 2017, ASD 1.104) outperformed more generalist multi-organ methods (MSD 2018, ASD 1.342) under challenging cases.
Tumor segmentation improved but remained limited Liver tumor Dice scores improved from 0.674 (ISBI 2017) to 0.702 (MICCAI 2017) to 0.739 (MICCAI 2018) on the same test set. In 2018, the winning nn-UNet approach (Isensee et al.) used a self-configuring 3D pipeline and significantly outperformed the third-place team. Tumor ASD also improved substantially from 1.189 to 0.903.
Tumor detection: small lesions remain the critical challenge A separate detection analysis found that best-performing segmentation algorithms did not necessarily excel at detecting individual lesions. The best lesion-wise recall was 0.554 (MICCAI 2018), with small lesions below 10 mm achieving F1 scores around 0.10. The best overall tumor burden score on the CodaLab leaderboard as of April 2022 was 82.5% Dice, versus an inter-rater variability of 95.2% between board-certified radiologists re-checking existing annotations.
Continuous improvement through open benchmark Year-by-year analysis of CodaLab submissions showed statistically significant improvement in mean Dice from 2021 to 2022, confirming that LiTS remains an active driver of methodological progress. The platform enables standardized comparison of new methods against the full historical record of submissions.
Small lesion segmentation is the primary bottleneck Performance was strongly stratified by tumor size. State-of-the-art methods performed well on large tumors but failed on lesions smaller than 10 mm, where the tumor may span only a few voxels against a background of noise and artifacts. Best results were achieved when volumes contained fewer than six tumors with total volume above 40 mm3; worst results occurred with single small sub-10 mm3 tumors.
Image contrast is a key determinant of accuracy Methods performed best when the tumor-to-liver Hounsfield unit (HU) difference was in the 40-60 HU range. Performance degraded sharply when contrast was below 20 HU - the range where many early-stage or iso-dense tumors fall. Hypo-dense lesions that are darker than the surrounding liver parenchyma were particularly difficult for current deep learning models.
Segmentation quality does not equal detection quality A top Dice score for tumor volume did not reliably predict good lesion-level detection. Some high-Dice methods generated excessive false positives (precision as low as 0.148), indicating that they segmented large tumor volumes accurately but also produced spurious detections in non-tumor areas. This highlights that Dice alone is insufficient for clinical evaluation, and detection metrics should be incorporated into future benchmarks.
Inter-rater variability reveals annotation uncertainty When a second radiologist re-annotated 15 CT scans from scratch, the median Dice agreement with the original annotations was only 70.2%. Board-certified radiologists reviewing and correcting existing annotations achieved 95.2% agreement. This range defines both the realistic ceiling for automated methods and the inherent ambiguity in ground truth - particularly for small and boundary-ambiguous lesions.
Stratified evaluation by lesion size and type Future benchmarks should evaluate performance separately by lesion size (small, medium, large), lesion type (primary vs. metastatic), and lesion-to-background contrast. Aggregating all cases into a single Dice metric obscures clinically meaningful differences and hides the failure modes that matter most in actual clinical deployment.
Multi-rater annotations to model uncertainty Single-rater annotations introduce label bias that is particularly acute for small and ambiguous lesions. Future datasets should collect annotations from multiple independent raters, enabling uncertainty-aware training and evaluation. Probabilistic segmentation models that output a distribution of plausible contours - rather than a single deterministic mask - would better reflect the inherent ambiguity in tumor boundary delineation.
Semi-automated labeling and federated learning Generating large labeled datasets requires substantial radiologist time. Advanced semi-automated annotation methods could accelerate dataset expansion. The multi-institutional structure of LiTS also makes it a natural test bed for federated learning, where models can be trained across institutions without sharing patient-level data - addressing legal and privacy barriers to large-scale multi-center collaboration.
Domain adaptation and open benchmarking Because LiTS training and test data share the same institutional distribution, generalizability to entirely new scanner types or acquisition protocols has not been tested. Future work using the LiTS data for domain adaptation research - where models must transfer across acquisition settings - would test real-world robustness. The CodaLab platform, with its open submission policy, will remain a community resource for benchmarking new segmentation methods as the field advances.