Deep learning models for medical imaging require enormous quantities of labeled training data to achieve robust performance. Yet most hospitals and research centers work in isolation, each holding datasets that are far too small to train a reliably generalizable model on their own. A model trained at one institution often performs significantly worse when applied at another -- a problem called poor generalizability.
This failure occurs because patient populations, imaging equipment, acquisition protocols, and even how radiologists label images differ between institutions. A model that learns patterns specific to one hospital's scanner type or patient demographics may not recognize the same disease features when it encounters data from a different setting.
Three traditional solutions exist but each has critical limitations. Centralized data pooling -- sending all data to a single location for joint training -- violates patient privacy regulations and institutional data governance policies. Public benchmark datasets are available but typically too small, too narrow in scope, and not representative of the full diversity of clinical practice. Transfer learning from publicly trained models helps but still does not capture institution-specific variation in data distribution.
The solution gaining momentum is distributed collaborative learning -- training AI models across multiple institutions simultaneously without any institution ever sharing its raw patient data. This paper in RadioGraphics provides a comprehensive educational overview of the methods, real-world examples, and remaining challenges of this approach for medical imaging AI.
Model ensembling is the simplest approach: each institution trains a completely independent model on its own data, and their predictions are combined -- either by averaging, majority vote, or weighted combination -- at inference time. No model weights are shared during training, making this highly privacy-preserving. However, it does not allow institutions to benefit from each other's data during training and can underperform when institutional datasets are very different in size or quality.
Federated averaging (FedAvg), the most widely studied method, works iteratively. Each institution trains the model locally on its own data for one round, then sends only the model's weights -- the numerical parameters encoding what the model has learned -- to a central coordinator. The coordinator averages these weights to create an improved global model, which is then sent back to all institutions for the next round. This process repeats until the model converges. Raw patient data never leaves any institution.
Split learning divides a neural network into two parts: the early layers remain at each institution and process local data, while the later layers are hosted centrally. Intermediate activations -- the outputs of the locally-held layers -- are sent to the central server for processing by the remaining layers. Only these activations, not raw images, are transmitted. This reduces computation requirements at each site but introduces a different form of partial data exposure.
Cyclic weight transfer passes a single shared model sequentially from institution to institution. Each institution trains the model for a defined period on its own data, then passes the updated model to the next institution in the cycle. Unlike federated averaging, there is no central coordinator -- the model migrates around the network. Swarm learning extends this concept by using blockchain-based peer-to-peer coordination, eliminating the need for any central server and distributing governance across all participating nodes.
Federated learning has been applied to breast density classification across multiple institutions. One federated model trained across sites achieved a 6.3% improvement in performance compared to models trained on data from a single institution. This improvement came from exposure to greater variation in imaging equipment and patient demographics -- precisely the diversity that single-institution training cannot provide.
In a landmark demonstration of federated learning at scale, a federated model was trained across 20 institutions in five countries to predict oxygen requirements in COVID-19 patients from chest X-rays. The federated model outperformed models trained at any individual site and showed far better generalizability when applied to data from new institutions not included in training. No institution shared a single patient record.
Swarm learning -- using blockchain-based coordination -- has been applied to four medical domains simultaneously: blood cancer (leukemia), tuberculosis, COVID-19 detection, and lung disease. A single swarm learning framework trained models across all four diseases, with each disease's model matching or exceeding the performance of a centrally trained model. The absence of a central server makes swarm learning particularly attractive for international collaborations where no single trusted coordinator exists.
In prostate cancer specifically, federated learning has been used to train a model for prostate gland segmentation from MRI across three institutions. The federated model matched the performance of a model trained on pooled data from all three sites -- validating that federation can achieve the same result as data sharing without the privacy cost. A larger federated brain tumor segmentation project across 10 institutions similarly achieved 99% of the performance of a centrally trained model.
Several open-source software frameworks make it practical for healthcare institutions to implement distributed collaborative learning without building infrastructure from scratch. FLARE (Federated Learning Application Runtime Environment), developed by Nvidia, is designed specifically for healthcare AI and supports federated averaging, differential privacy, and secure aggregation out of the box.
OpenFL, developed jointly by Intel and the University of Pennsylvania, is another mature framework with specific support for medical imaging workflows. It was used in the 10-institution brain tumor segmentation project mentioned above, demonstrating its readiness for large-scale clinical AI collaboration.
Community-driven frameworks include Flower (developed by Adap), which prioritizes flexibility and supports a wide range of machine learning libraries; PySyft, which emphasizes privacy-preserving technologies including differential privacy and secure multi-party computation; and TensorFlow Federated, Google's official federated learning extension for its popular TensorFlow AI framework. Fed-BioMed is designed specifically for biomedical research use cases.
The availability of these frameworks significantly lowers the technical barrier to collaborative learning. Institutions do not need to implement federated averaging algorithms from scratch -- they can integrate existing frameworks with their own data pipelines and institutional authentication systems. However, establishing the data governance agreements, institutional review, and security protocols required for multi-institutional collaboration remains a significant non-technical challenge.
Even when institutions collaborate, their datasets are not identical. Distribution shift refers to systematic differences between institutions' data that can prevent a shared model from performing equally well at all sites. Three types of distribution shift are particularly relevant in medical imaging AI.
Covariate shift -- also called feature shift -- occurs when the statistical properties of the input images differ between institutions, even if the underlying diseases are similar. Different scanner manufacturers, acquisition protocols, contrast agents, and imaging resolutions all contribute to covariate shift. A model that has only seen images from one scanner type may not recognize the same anatomical structures when they appear slightly different due to acquisition differences.
Label shift occurs when the distribution of diagnoses or outcomes differs between institutions, often reflecting genuine differences in patient populations. An institution serving an older or higher-risk population will see more advanced cancers than one serving a screening population. A federated model that aggregates data without accounting for this will be biased toward the most common presentation across all sites, potentially underperforming at institutions with unusual case mixes.
Concept shift -- the most subtle form -- occurs when the relationship between imaging features and clinical outcomes changes between institutions, even when input and label distributions are similar. This can arise from differences in how pathologists define disease grades, how radiologists describe lesions, or how clinical endpoints are measured. Detecting and correcting concept shift requires careful cross-institutional calibration studies and is an active area of research in distributed medical AI.
Sharing model weights instead of raw data is not perfectly privacy-preserving. Two classes of attacks have been demonstrated against federated learning systems. Model inversion attacks attempt to reconstruct the original training images from the model weights that were shared. Researchers have demonstrated that in some settings, surprisingly detailed images can be partially recovered from gradient updates, particularly when training datasets are small.
Membership inference attacks attempt to determine whether a specific patient's data was used to train a model. By probing the model with carefully chosen queries, an attacker can often determine with better-than-chance accuracy whether a particular individual was in the training set -- potentially revealing sensitive information about that person's medical history.
Several cryptographic defenses can mitigate these risks. Differential privacy adds carefully calibrated statistical noise to model updates before they are shared, making it mathematically provable that no individual's data can be identified from the shared weights. However, this noise degrades model accuracy and the optimal privacy-utility tradeoff must be tuned for each application. Homomorphic encryption allows mathematical operations to be performed on encrypted data without decrypting it first, so model aggregation can occur without the coordinator ever seeing unencrypted weights.
Secure multi-party computation (SMPC) distributes the computation across multiple parties such that no single party ever sees the complete information needed to reconstruct private data. Confidential computing uses hardware-level security -- specialized processor chips called Trusted Execution Environments -- to ensure that computation occurs in an isolated enclave that cannot be accessed even by the server's operating system. These approaches can be combined, though each adds computational overhead and implementation complexity.
A federated model that achieves high average performance across all institutions may still perform poorly for specific demographic subgroups or at institutions serving underrepresented populations. This is called performance fairness -- and it is a documented concern in medical AI, where models trained predominantly on data from large academic centers may underperform when deployed at community hospitals or institutions serving minority populations.
A distinct but related concern is collaboration fairness -- ensuring that institutions that contribute more data or higher-quality data receive proportional benefit from the collaborative model. If small institutions contribute data without receiving a better model in return than they could have trained alone, they have little incentive to participate. Approaches based on Shapley values -- a mathematical framework borrowed from cooperative game theory -- can quantify each institution's marginal contribution to the collaborative model and guide equitable benefit-sharing arrangements.
These fairness concerns have important implications for healthcare equity. If federated learning systematically advantages large, data-rich institutions over smaller community providers, it could amplify existing disparities in AI-assisted care quality. Researchers argue that fairness should be a first-class design criterion for collaborative learning systems, not an afterthought addressed once a model is already deployed.
Achieving fairness in federated learning requires both technical solutions -- such as federated optimization algorithms that explicitly weight performance across subgroups -- and governance solutions -- such as data-sharing agreements that define performance standards all participating institutions must meet before a model can be deployed at any site.
Distributed collaborative learning represents a genuine solution to one of the central challenges in medical AI: the need for large, diverse training datasets in a regulatory environment that prohibits centralized patient data sharing. The five methods described -- ensembling, federated averaging, split learning, cyclic weight transfer, and swarm learning -- offer a spectrum of privacy-performance tradeoffs suited to different collaboration structures and institutional capabilities.
Real-world deployments have validated that federated models can match or approach the performance of models trained on pooled data, even across dozens of institutions in multiple countries. The 99% performance retention achieved in the 10-institution brain tumor segmentation project, and the performance improvements seen in breast density classification and COVID-19 outcome prediction, demonstrate that federation is not merely a theoretical privacy solution but a practical performance tool as well.
Significant challenges remain. Distribution shift, privacy attacks, computational overhead from cryptographic defenses, and fair governance frameworks each require ongoing research and standardization. The technical infrastructure for federation is increasingly available through open-source frameworks, but the institutional, legal, and ethical infrastructure for safe, equitable collaboration is still being developed.
For radiologists and medical imaging researchers, the practical message of this review is that the barriers to participating in multi-institutional AI collaboration are falling. Institutions do not need to share a single patient image to contribute to and benefit from models trained on far larger and more diverse data than any single center could assemble alone. As these methods mature, they are likely to become the standard approach for training clinically deployable medical imaging AI.