The Problem with Standard AI Training Supervised learning (SL) models for medical diagnosis are typically trained to maximize overall accuracy or average recall, treating all diagnostic errors equally. In clinical practice, however, missing a melanoma carries far greater consequences than a false positive biopsy. Human preferences about these relative harms - which are patient-specific, cancer-type-specific, and context-dependent - are rarely incorporated into AI training, but are instead addressed post-hoc through operating point thresholds.
Reinforcement Learning as a Solution Reinforcement learning (RL) frames diagnosis as a sequential decision-making problem: the AI takes actions (diagnoses, management decisions) and receives rewards or penalties based on expert-defined tables that quantify the relative harm of different error types. By training with unequal rewards (high penalty for missed melanoma, moderate penalty for unnecessary excision), the RL model learns a policy that directly optimizes clinically meaningful outcomes rather than statistical accuracy.
Key Results Compared to the supervised learning baseline, the RL model improved melanoma sensitivity from 61.4% to 79.5% and basal cell carcinoma sensitivity from 79.4% to 87.1% while maintaining overall accuracy. In a reader study with 89 dermatologists, RL model support increased the rate of correct diagnoses by 12.0% and optimal management decisions from 57.4% to 65.3%.
Supervised Learning Baseline A SL model was trained on 10,015 dermoscopic images (publicly available training set) to classify 7 skin conditions: melanoma (MEL), basal cell carcinoma (BCC), actinic keratosis/intraepidermal carcinoma (AKIEC), benign keratinocytic lesion (BKL), melanocytic nevus (NV), dermatofibroma (DF), and vascular lesion (VASC). Training minimized class-frequency weighted cross-entropy loss to maximize average recall. On an independent test set of 1,511 images, the SL model achieved 77.8% average accuracy with 61.4% melanoma sensitivity.
Expert-Generated Reward Table Dermatologists generated a 7x7 reward-penalty table specifying numeric scores for each combination of true diagnosis and predicted diagnosis. Missed melanomas received the highest penalty (reward = -5), while correctly identifying melanoma received the highest reward (+5). Misclassifying a melanoma as nevus received a penalty of -5, while misclassifying it as BCC received -3. This table encodes asymmetric clinical costs and can vary per clinical scenario.
Deep Q-Learning RL Architecture The RL model used deep Q-learning. The initial state was a one-dimensional vector combining the SL model's multiclass probability output and the SL model's feature vector. The RL agent learned an action-selection policy that maximized cumulative reward (from the expert reward table) using the same training and test sets as the SL model. Shannon entropy was used to measure prediction uncertainty.
Sensitivity Improvements On the 1,511-image test set, the RL model achieved melanoma sensitivity of 79.5% (95% CI: 73.5-85.6%) versus 61.4% for the SL model (P < 0.001), and BCC sensitivity of 87.1% (95% CI: 80.3-93.9%) versus 79.4% (P < 0.001). Overall accuracy increased slightly from 77.8% to 79.2%. The melanoma sensitivity gain was primarily driven by reclassifying melanomas that the SL model had incorrectly labeled as nevi.
Reduced Overconfidence The RL model increased Shannon entropy (uncertainty) of predictions: SL model median entropy 0.30 bits versus RL model median 1.46 bits (P < 0.001). Crucially, this increased uncertainty specifically affected incorrect predictions - the RL model's incorrect predictions had higher entropy than the SL model's incorrect predictions (1.81 bits vs. 1.13 bits, P < 0.001). For correctly predicted melanomas, RL model uncertainty decreased, indicating more confident correct diagnoses and less confident incorrect ones.
Confusion Matrix Analysis The SL confusion matrix showed melanoma being frequently confused with nevus (22% of melanomas classified as NV). The RL confusion matrix substantially reduced this melanoma-to-nevus misclassification while maintaining good specificity for benign conditions. Benign condition sensitivity did not substantially decrease with RL model use, demonstrating that sensitivity improvements for malignant lesions did not come at the cost of benign misclassification.
Reader Study Design 89 dermatologists diagnosed skin lesions without AI, with SL model support, or with RL model support. For each lesion, they selected one of four management decisions: dismiss, excise, treat locally, or monitor. Optimal management was defined as excision for melanoma and BCC, local therapy for AKIEC, and dismiss for benign conditions.
Diagnosis Accuracy Improvement Correct diagnosis rate without AI was 68.0% (95% CI: 65.3-70.6%). With SL support it increased to 75.3% (mean difference +7.3%, P < 0.001) and with RL support to 79.9% (mean difference +12.0%, 95% CI: 8.8-15.1%, P < 0.001). Melanoma sensitivity improved from 62.4% without support to 69.4% with SL support and to 83.9% with RL support (P < 0.001 for RL vs. no AI).
Optimal Management Improvement The proportion of optimal management decisions increased from 57.4% without AI to 61.7% with SL support (P = 0.03) and to 65.3% with RL support (95% CI: 61.7-68.9%, P < 0.001). For melanoma specifically, the optimal management rate was 70.1% without AI, 73.4% with SL support, and 86.4% with RL support (P < 0.001 for RL vs. no AI).
Binary Scenario (Malignant vs. Benign) In a simplified two-class (malignant/benign) with two actions (dismiss/excise) scenario, the naive model (excise if malignancy probability > 0.5) achieved 78.2% TPR for malignancies. Both the threshold-adjusted SL model (88.9%) and the RL model (88.0%) substantially outperformed the naive approach. For melanoma specifically, TPR improved from 68.4% (naive) to 85.4% (threshold) and 82.5% (RL), with no significant difference between threshold and RL approaches in this simpler setting.
Multiclass Scenario (7 Classes, 3 Actions) In the full 7-class setting with three management options (dismiss, excise, local therapy), the RL model and threshold model both outperformed the naive approach. For the 307 malignant conditions requiring treatment, mismanagement was 21.8% for naive, 5.2% for threshold-adjusted SL (best performance), and 13.4% for RL. In this scenario the threshold-adjusted model performed better, suggesting RL offers greater benefit in more complex, patient-centered settings.
Patient-Centered Monitoring Scenario In the most complex scenario involving 7,375 lesions from 524 high-risk patients with multiple nevi (including 55 noninvasive or microinvasive melanomas), the RL model detected 61.8% of melanomas (vs. 9.1% naive, 25.5% threshold at a standard cutoff), monitored an additional 20% of melanomas acceptably, and limited excess benign excisions. The RL model uniquely captured the clinical logic of monitoring suspicious nevi rather than excising all uncertain lesions.
Retraining Requirement A key limitation of RL compared to threshold-based approaches is that the RL model must be retrained when reward tables change, while threshold adjustments can be applied at inference time without retraining. For clinical scenarios requiring individualized reward tables per patient or clinical context, this creates a deployment challenge. The authors note that both threshold and RL methods outperform the naive model in most scenarios, suggesting threshold approaches may be sufficient for simpler applications.
Patient Preferences Not Included The current reward tables reflect physician preferences only. Patient-centered care requires incorporating patients' own risk-benefit assessments, which can differ substantially from physician assessments - for example, patients may weight the anxiety of monitoring differently than physicians. Future systems should enable collaborative development of reward tables through shared decision-making between patients and clinicians.
Transparency and Acceptance Reward tables provide an underutilized secondary benefit: they make the trade-offs encoded in AI systems explicit and auditable, potentially increasing clinician and patient trust and regulatory acceptance. The authors suggest this transparency advantage - knowing exactly what harms and benefits the AI is optimizing for - could be important for AI governance in clinical settings. Future research should evaluate provider and patient satisfaction with RL-based decision support and extend the approach to other diagnostic imaging domains beyond skin cancer.