The Clinical Question AI systems have demonstrated performance equal to or exceeding dermatologists in controlled dermoscopic melanoma classification studies. However, demonstrating that an AI can match experts on its own does not answer the clinically relevant question: does providing AI output as real-time decision support actually improve expert diagnostic performance? This study directly investigated whether CNN-based AI assistance improves dermatologist accuracy.
Study Design Twelve board-certified dermatologists from 9 German university hospitals each classified 100 unique biopsy-verified dermoscopic images (50 melanomas, 50 nevi) twice: first using personal expertise alone (Part I), then again with CNN-generated classification and confidence level displayed alongside each image (Part II). This within-participant design allowed direct comparison of each dermatologist's performance with and without AI support on identical images.
Key Finding AI support significantly improved dermatologist sensitivity from 59.4% to 74.6% (P = 0.003) and overall accuracy from 65.0% to 73.6% (P = 0.002), while specificity remained nearly unchanged (70.6% to 72.4%, P = 0.54). The CNN alone achieved 84.7% sensitivity and 81.9% accuracy on the same images, outperforming the human-AI combination.
CNN Training Images were sourced from the ISIC archive, predominantly from the HAM10000 dataset, selecting only biopsy-verified melanomas (n=1,633) and nevi (n=3,311). For each of the 24 test subsets (12 participants x 2 surveys each), a separate training set of 4,894 images was constructed by removing the test images. Class imbalance was corrected by duplicating melanoma images and applying online augmentation. The classifier performance was characterized on a validation set before the study (mean sensitivity 78%, specificity 81%, accuracy 80%).
Survey Protocol Each dermatologist received a unique set of 100 images split into two 50-image surveys for Part I and two identical 50-image surveys for Part II. Part I surveys were completed before any Part II surveys. In Part I, each image was shown with three questions: image quality rating, benign/malignant classification, and confidence (0-10 scale). In Part II, the same images were shown with the dermatologist's Part I answer visible along with the CNN's binary classification and its confidence percentage.
Statistical Analysis The primary endpoint was accuracy with AI versus without AI (Wilcoxon test). Secondary endpoints included sensitivity and specificity changes. Significance threshold was P < 0.016 (Bonferroni-corrected for primary and two secondary endpoints). A learning effect analysis compared performance on survey 1 versus survey 2 within each study part using Pearson's chi-squared test.
Classification Performance Summary Without AI: mean sensitivity 59.4%, specificity 70.6%, accuracy 65.0%. With AI support: sensitivity 74.6%, specificity 72.4%, accuracy 73.6%. The CNN alone: sensitivity 84.7%, specificity 79.1%, accuracy 81.9%. All 12 dermatologists individually experienced improved balanced accuracy with AI support, with lower-performing physicians showing the largest absolute gains.
Switching Behavior Analysis Dermatologists and CNN agreed in 63% of cases (55% correct, 8% both incorrect). In the 27% of cases where the CNN was correct but the dermatologist was not, dermatologists switched to the correct answer 46% of the time. In the 10% of cases where the dermatologist was correct but the CNN was not, dermatologists switched to the wrong answer 39% of the time. Dermatologists showed a 2.6-fold higher willingness to switch from benign to malignant (72%) than from malignant to benign (28%), likely reflecting risk aversion around missed melanomas.
Confidence Changes with AI Dermatologist confidence increased from 67.0% to 79.1% when they agreed with the CNN (regardless of correctness) and decreased from 61.7% to 44.3% when they disagreed. This dual effect is important: agreement reinforced confidence even when both were wrong, while disagreement appropriately reduced confidence but also sometimes caused dermatologists to abandon correct diagnoses. CNN confidence was well-calibrated: 71.7% when correct versus 34.8% when incorrect.
No Learning Effect Without AI Comparison of dermatologist performance between the first and second surveys within Part I showed no significant change in sensitivity (P = 0.50) or specificity (P = 0.76), confirming that the test-retest design did not introduce a systematic learning bias from repeated image viewing alone.
Trend Toward Learning with AI In Part II, dermatologists tended to perform better on their second AI-supported survey compared to their first, suggesting they learned to better integrate CNN outputs into their diagnostic reasoning after one practice session. In divergent cases (where CNN and dermatologist disagreed), the switching rate at correct moments increased from 38% in survey 1 to 49% in survey 2. These learning differences did not reach statistical significance (sensitivity P = 0.21, specificity P = 0.43).
CNN Confidence as a Switching Predictor Among cases where dermatologists switched their diagnosis in response to disagreeing with the CNN, the mean CNN confidence was 63.3% compared to 53.7% for cases where no switch occurred. This suggests dermatologists were appropriately more responsive to high-confidence AI disagreements, though refining this trust calibration further could improve outcomes.
Sensitivity Gain Without Specificity Loss The primary finding - improved sensitivity at nearly constant specificity - is clinically meaningful because melanoma prognosis is strongly stage-dependent, making false negatives more costly than false positives in image-based screening. The increase in sensitivity suggests that AI assistance specifically helps dermatologists identify melanomas they would have otherwise missed.
Human-AI Combination vs. AI Alone The CNN alone outperformed the dermatologist-AI combination (accuracy 81.9% vs. 73.6%). This gap partly reflects the artificial survey setting - dermatologists in real practice integrate additional information including patient age, medical history, lesion localization, and clinical context not available in isolated image classification tasks. The study authors caution against fully autonomous AI diagnosis, noting that patients prefer AI systems that are interpreted by physicians rather than replacing them.
The Overconfidence Risk A key concern is that confidence increased even when both the dermatologist and AI were simultaneously wrong (both incorrect group). This AI-reinforced overconfidence could potentially delay recognition of a misclassified lesion. Future AI interfaces should include uncertainty calibration features that signal when neither the AI nor the prior human judgment should be trusted highly.
Artificial Setting Limitations The binary classification task (melanoma/nevus only) and the use of biopsy-verified edge-case images in a 50:50 class balance do not reflect real clinical practice where melanoma prevalence is low and clinical context informs diagnostic decisions. In practice, dermatologists likely apply a different decision threshold, excising uncertain lesions rather than committing to a classification. Post-hoc analysis showed that converting low-confidence nevus diagnoses to excision recommendations increased effective sensitivity.
Small Participant Sample Only 12 board-certified dermatologists participated, all of whom were coauthors of the paper and academic specialists. The sample does not include trainee dermatologists or general practitioners, who might respond differently to AI support. Larger, more diverse participant cohorts would be needed to generalize the conclusions to real-world clinical use.
Required Next Steps Clinical trials integrating AI classifiers into actual dermoscopy workflows - where physicians also have access to patient history and dermoscopic clinical context - are the critical next step. Future classifiers should distinguish multiple lesion classes beyond melanoma/nevus, include robust uncertainty estimation to guide trust calibration, and be evaluated in prospective real-world settings to assess actual patient outcome impact.