Deep learning survival (DLS) models are a type of artificial intelligence trained to predict how long a patient might live after a cancer diagnosis. Unlike simpler models that classify patients as just high- or low-risk, survival models generate a continuous risk score that ranks patients along a spectrum, providing more nuanced prognostic information.
While these models show great promise, a critical gap exists: most studies develop AI models at a single institution and test them only on similar data from that same institution. When such models are then applied to patients from a different hospital, using a different CT scanner with different image settings, they often perform poorly. This multi-institutional generalizability problem is one of the biggest barriers to clinical adoption of AI in cancer care.
This study takes a systematic approach to understanding what training decisions most affect whether a survival AI model generalizes to new institutions. Rather than just reporting one model's performance, the researchers ran hundreds of carefully controlled experiments to identify which choices in data handling, model setup, and image processing make the biggest difference for real-world reliability.
The study assembled a dataset of 525 kidney cancer patients from 9 different institutions, drawing from four publicly available repositories: TCGA-KIRC, TCGA-KICH, TCGA-KIRP (kidney cancer datasets from The Cancer Genome Atlas), and KiTS19 (a kidney tumor segmentation challenge dataset). Combining data from 46 different CT scanner models spanning 9 institutions creates exactly the kind of real-world variability that challenges AI models.
All models used a 3D ResNet-18 architecture trained with a Cox loss function, which is designed specifically for survival analysis. This combination is one of the most established approaches in the field, making the results broadly relevant rather than specific to an exotic new method. The models were evaluated using two standard metrics: concordance index (c-index), which measures how accurately the model ranks patients by survival time, and hazard ratio (HR), which quantifies how different the survival outcomes of high-risk and low-risk groups are.
Three controlled experiments were designed to isolate the effect of specific training choices: (1) random versus intelligent data partitioning using the CohortFinder algorithm, (2) fixed versus randomized model initialization and data ordering, and (3) different types and intensities of image augmentations. Running 200 to 500 independent models per experiment allowed statistical comparison of each approach.
The most dramatic finding came from the data partitioning experiment. When patients were divided into training, testing, and validation sets using the standard approach of random assignment, the model performed well on training and testing but nearly completely failed on external validation, achieving a c-index of only 0.56 (barely better than chance) and a hazard ratio of 0.29 (effectively no separation between risk groups).
By contrast, when the CohortFinder intelligent partitioning algorithm was used to ensure all three data splits proportionally represented each of the 9 institutions and scanner types, the external validation c-index jumped to 0.74 and the hazard ratio improved to 5.08. This means high-risk patients identified by the model had more than 5 times the event rate of low-risk patients, a clinically meaningful separation.
This comparison powerfully illustrates a fundamental problem with how many AI medical models are currently developed: random data splitting creates an illusion of performance that collapses when the model faces data from a new institution. Institution-aware data splitting is not just an improvement; it is essential for building models that will actually work in clinical practice.
The second experiment revealed a clear separation between two training factors. Model initialization, meaning how the model's internal parameters are set before training begins, had a significant impact on both performance and variability across independent runs. Models with fixed (reproducible) initialization consistently outperformed those with random initialization and showed much less variation between runs.
By contrast, data ordering, meaning the sequence in which training samples are presented to the model during each training epoch, had little to no effect on performance. Changing from fixed to random data ordering resulted in less than a 1.5% difference in validation c-index. This is a practically important negative finding: researchers do not need to worry about optimizing data order, freeing them to focus their effort on more impactful factors.
The finding about initialization has a practical implication: when reporting AI model results, researchers should train multiple independent models with different random seeds and report the distribution of results, not just the best-performing single run. This replicate training approach gives a much more honest picture of how a model would perform when deployed.
Image augmentations are transformations applied to training images to help the AI learn features that generalize across different scanning conditions. The study tested three types: intensity transforms (shifting image contrast to simulate scanner differences), spatial transforms (rotating and warping images to simulate patient positioning differences), and Gaussian noise (adding random pixel variation to simulate scanner noise).
Combining all three augmentation types produced the best overall external validation performance, improving c-index by +4.76% and hazard ratio by +44.39% compared to no augmentation. The single most impactful individual augmentation was additive Gaussian noise, which improved c-index by +3.17% and hazard ratio by +23.47% on external validation, suggesting that teaching the model to tolerate scanner noise is particularly important for cross-institutional generalization.
An interesting nuance emerged regarding augmentation intensity: high-intensity augmentations during training sometimes reduced performance on the training set while improving validation performance, the opposite of what might be expected. This reflects the regularization effect of augmentation: making training harder forces the model to learn more robust and generalizable features rather than memorizing the specific characteristics of the training images.
This study provides the most comprehensive empirical investigation to date of the training choices that determine whether AI survival models will actually work when applied at a new institution. The three key takeaways are: use institution-aware data partitioning, use fixed model initialization, and apply diverse image augmentations including Gaussian noise.
The findings apply broadly beyond kidney cancer to any AI survival model developed using multi-institutional medical imaging data. The methodology of running hundreds of replicate models and using statistical testing to compare configurations represents a rigor standard that many published AI medical studies currently lack.
Ultimately, this work helps answer a question that matters enormously for patients: can AI tools developed in academic research centers actually be trusted to work in community hospitals with different equipment? The answer is yes, but only when developers make specific, deliberate choices in how they build and validate their models, choices that this study now clearly identifies.