The clinical problem Early-stage melanoma patients face a difficult treatment decision: adjuvant immunotherapy can prevent recurrence but causes significant toxicity and costs over $20,000 per month per patient. At ten years, death rates range from 2-8% for stage I, 12-25% for stage II, and 12-40% for stage III disease. Better prognostic tools are needed to identify which patients truly need treatment versus who can safely forgo it.
A deep learning solution Kulkarni, Robinson, and colleagues developed a deep neural network (DNN) that predicts disease-specific survival (DSS) and distant metastatic recurrence (DMR) from standard H&E-stained melanoma slides - the same tissue specimens already prepared for routine pathological diagnosis. The model requires no additional tissue processing, RNA extraction, or immunohistochemical staining.
Validated performance The model was trained on 108 patients from four institutions, then validated on 104 patients from Yale School of Medicine (YSM, AUC 0.905, p < 0.0001) and independently tested on 51 patients from Geisinger Health Systems (GHS, AUC 0.880, p < 0.0001). The same cutoff developed in the YSM cohort predicted DSS in the GHS cohort on Kaplan-Meier analysis (p < 0.0001).
The challenge of large pathology images Whole-slide melanoma images are 1-15 gigabytes each - far too large for direct neural network processing. Standard approaches of dividing images into small patches fail for prognosis prediction because recurrence risk depends on tumor-immune cell interactions across a spatial region, not just local image features in individual patches.
Cell-type-sensitive sampling strategy The pipeline begins by using QuPath digital pathology software to segment and classify all cell nuclei in the slide into three categories: tumor cells, immune cells, and other. The slide is divided into tiles, and only tiles with minimum tumor and/or lymphocyte density are retained. From these relevant tiles, spatially localized sequences of 20 image patches are randomly generated, creating RNN input sequences that capture the spatial context of tumor-immune interactions.
CNN-RNN combined architecture Each 500x500 pixel patch (downsampled to 100x100) is processed by the CNN, which extracts high-level image features. These CNN feature vectors are then processed sequentially by the RNN, which identifies discriminative spatial patterns across the sequence. The RNN output is combined with pre-computed morphological features (nuclear size, immune cell density and distribution) through fully connected layers to generate a per-sequence recurrence probability vote, which is aggregated across all sequences for a final patient-level prediction.
The most important feature: lymphocyte-to-tumor ratio To assess which features drove prediction accuracy, the researchers ran ablation experiments, removing each feature individually and measuring the impact on AUC. Removing the ratio of lymphocyte area to tumor cell area caused the largest accuracy drop, from AUC 0.880 to 0.509 (essentially random). However, using this feature alone in QuPath also failed to predict DMR (AUC 0.589), demonstrating that immune cell content is necessary but not sufficient - it must be integrated within the deep learning context.
Combination of features is essential The classifier relies on multiple features working in combination: nuclear size ratios, immune cell density and clustering (via Delaunay triangulation), spatial distribution of lymphocytes relative to tumor cells, and the raw image pixel data captured by the CNN. No individual feature was independently predictive, but their combination achieved the high AUC values. This mirrors clinical pathology practice where experienced pathologists integrate multiple findings.
Institution-independent performance The training cohort included patients from four different institutions (Columbia University, NYU Medical Center, Geisinger Health System, Icahn School of Medicine at Mount Sinai), and the model demonstrated robust performance across institution-specific staining variations. This cross-institutional generalizability is critical for eventual clinical deployment.
Yale School of Medicine validation The first validation cohort of 104 YSM patients (after 14 exclusions from 118 screened) achieved AUC 0.905 for DMR prediction. A single cutoff was selected in this cohort to maximize sensitivity for recurrence - specifically designed to identify patients who could be excluded from clinical trials by their low recurrence risk. Kaplan-Meier analysis showed highly significant separation of DSS curves (p < 0.0001), and multivariable analysis confirmed the DNN predictor added independent value beyond standard clinical variables.
Geisinger Health Systems validation The second validation cohort of 51 GHS patients (after 5 exclusions from 56 screened) achieved AUC 0.880. Applying the same cutoff determined from the YSM cohort to this entirely independent dataset, 24 patients were classified as favorable (5 developed DMR, 19 did not) and 27 were classified as unfavorable (24 developed DMR, 3 did not). KM analysis showed highly significant DSS separation (p < 0.0001) and multivariable analysis confirmed independence from standard clinical predictors (P < 0.001).
Endpoint selection rationale The DMR endpoint was deliberately chosen over overall survival because the advent of effective immunotherapy since 2017 has made survival increasingly treatment-dependent, reducing its consistency as a reflection of underlying tumor biology. DMR captures the inherent aggressiveness of the primary tumor more reliably across the treatment era. The corresponding biomarker training on pre-2017 data thus captures biology rather than treatment effects.
Addressing the adjuvant therapy overtreatment problem Current AJCC staging does not capture the full complexity of recurrence risk: stage IIIA patients actually have better survival than stage IIC patients, depth estimates can be inaccurate with shave biopsies, and complete staging requires invasive lymph node surgery that provides no survival benefit. A computationally-derived biomarker from the primary H&E slide could replace or supplement these invasive and imperfect approaches.
Enriching clinical trial populations The cutoff was specifically designed to identify patients unlikely to benefit from adjuvant therapy who could be safely excluded from clinical trials. This has two benefits: it reduces unnecessary treatment exposure and toxicity for low-risk patients, and it enriches clinical trials with high-risk patients who are most likely to show a treatment benefit, increasing statistical power and accelerating drug development.
Practical advantages over existing biomarkers Existing prognostic tools for early-stage melanoma include genomic signature assays (NanoString), specialized RNA sequencing methods, and multiplexed immunofluorescence (QIF) assays. These require additional tissue processing, specialized equipment, and complex analyses not available in most clinical labs. The DNN biomarker requires only a digital scan of the standard H&E slide that is already prepared for routine pathological review, dramatically reducing barriers to clinical adoption.
Retrospective cohorts and small validation sets Both validation cohorts are retrospective, with a combined total of fewer than 200 patients. Retrospective data collection introduces potential selection bias, and the model has not been tested in the prospective setting where slides are acquired and analyzed in real time as part of clinical care. Prospective validation is essential before the biomarker can be incorporated into clinical decision-making.
High melanin content exclusions Approximately 10-15% of cases were excluded from analysis due to excessive melanin content obscuring cellular features. Highly pigmented melanomas represent a specific subpopulation that the current algorithm cannot evaluate. Slide bleaching techniques to reduce melanin interference, followed by validation of biomarker accuracy in this subset, would extend the applicability of the method.
Future directions The authors advocate for prospective validation in larger multi-institutional cohorts, with the ultimate goal of integrating the biomarker into AJCC staging criteria or adjuvant therapy selection guidelines. Additional areas for development include: testing performance across different melanoma subtypes (acral, mucosal), evaluating whether the biomarker predicts response to specific adjuvant therapies (not just recurrence risk), and automating the pipeline for integration with clinical digital pathology workflows.