Malignant lymphoma is among the most diagnostically complex areas in oncology, comprising approximately 100 distinct subtypes with overlapping morphological features. In current clinical practice, definitive subtype classification requires integrating multiple data streams: H&E-stained tissue specimens examined under a microscope, immunohistochemistry panels, cytogenetic testing, and flow cytometry (FCM). This multi-step process is resource-intensive and relies heavily on expert hematopathologist judgment, creating bottlenecks particularly in institutions without dedicated lymphoma pathologists on staff.
The role of flow cytometry: FCM is a single-cell measurement technique that attaches fluorescent antibody markers to cells in a collected specimen and measures optical properties of each cell as it passes through a laser beam. In lymphoma pathology, FCM results are provided to pathologists alongside H&E slides as a routine part of the diagnostic workup. The data are post-processed by laboratory technologists and delivered in binary format, indicating whether abnormal cell populations are present for each of a panel of antibody markers. Critically, FCM is most informative at the level of major lymphoma categories, or "superclasses," such as B-cell lymphoma, T-cell lymphoma, and Others, rather than at the level of individual subtypes.
The challenge for machine learning: Existing AI methods for lymphoma digital pathology primarily rely on whole slide image (WSI) analysis alone. While WSI-based approaches have achieved strong performance on binary classification tasks, multi-class subtype classification from images alone remains challenging. The authors hypothesize that incorporating FCM data can improve classification accuracy by mimicking the hierarchical diagnostic reasoning process used by pathologists: first determining the superclass from FCM evidence, then using tissue image features to distinguish among the subtypes within that superclass.
This paper, published in the Journal of Pathology Informatics in 2024, introduces a framework that formalizes this two-level reasoning into a single end-to-end trainable model using a Mixture of Experts (MoE) architecture combined with Multiple Instance Learning (MIL) for WSI analysis. The 600-case dataset covers six classes across three superclasses, with 100 cases per class, providing a controlled experimental setting to evaluate the contribution of each component.
The experimental dataset was drawn from a private clinical collection diagnosed at Kurume University, comprising 600 cases with definitive diagnoses according to WHO classification, confirmed by an expert hematopathologist using immunohistochemical staining and genetic testing. The six classes are organized into three superclasses: B-cell lymphoma (diffuse large B-cell lymphoma, DLBCL, and follicular lymphoma, FL), T-cell lymphoma (angioimmunoblastic T-cell lymphoma, AITL, and adult T-cell leukemia/lymphoma, ATLL), and Others (classical Hodgkin lymphoma, CHL, and reactive lymphoid hyperplasia, RL). Each class contains exactly 100 cases, deliberately balanced to enable robust initial evaluation of the proposed method.
Tissue digitization: H&E-stained tissue specimens were digitized using an Aperio GT450 scanner (Leica Biosystems) at 40x magnification (0.26 micrometers per pixel), producing WSIs with dimensions on the order of 100,000 by 100,000 pixels. Because these images are far too large to be directly input to neural networks, the processing pipeline extracts 224 by 224-pixel patches from tissue regions, with glass regions excluded via Otsu's binarization thresholding on average chromaticity. Each WSI yields 100 or more tissue patches for model input.
FCM data format: FCM measurements from 18 antibody markers were used, with B-cell-associated markers (CD10, CD19, CD20, CD23, kappa, lambda), T-cell-associated markers (CD2, CD3, CD4, CD5, CD7, CD8), and other markers (CD11c, CD16, CD25, CD30, CD34, CD56). For each case, the 18-dimensional FCM input vector is binary, with a value of 1 indicating confirmed presence of an abnormal cell population for that marker and 0 otherwise. This structured representation was processed by laboratory technologists before being provided to the model.
The reliability problem: A key challenge with FCM data is that abnormal cell populations are not always detectable even when the correct lymphoma type is present. In the dataset, 87 of 100 DLBCL cases and 96 of 100 FL cases show no abnormal cells by FCM, meaning that relying on FCM alone would systematically misclassify many B-cell lymphoma cases as belonging to the Others superclass (which is characterized by FCM negativity). Similarly, more than half of AITL cases show no T-cell-related abnormal populations. This variability is exactly the problem the multimodal gating mechanism is designed to address.
The proposed model consists of four functional components that together implement a multimodal, hierarchically-aware classification pipeline. The model takes as input a bag of multiple image patches extracted from a single WSI along with the case's 18-dimensional FCM binary vector, and produces a six-class probability prediction. The four components are: (i) feature extractors for images and FCM data, (ii) a multimodal gating network, (iii) multiple expert sub-networks specialized for each superclass, and (iv) an attention-based MIL aggregation module.
Feature extractors: Image patches are processed by a ResNet50 convolutional neural network pre-trained on ImageNet, producing a 2,048-dimensional feature vector per patch after global average pooling. FCM data is transformed by a two-layer multi-layer perceptron (MLP) into a 128-dimensional feature vector using ReLU activations. The FCM feature vector is replicated for each image patch in the bag so that it can be concatenated with individual patch features during the gating step.
Multimodal gating network: The central innovation is a gating network that receives both image and FCM features simultaneously. For each image patch, the gating network concatenates the 2,048-dimensional image feature with the 128-dimensional FCM feature and passes the result through a three-layer MLP with 512 hidden units, outputting a three-dimensional weight vector through a softmax activation function. This weight vector assigns a probability to each of the three expert sub-networks for that specific patch. A temperature parameter T controls the sparsity of these weights: lower temperatures (T=0.5) push weights toward hard selection of a single expert, while higher temperatures (T=0.8) produce softer, more distributed weights. The key advantage over using FCM alone for gating is that the image features compensate for FCM failures in cases where abnormal cell populations are not detected.
Expert sub-networks: Three sub-networks correspond to the three superclasses (B-cell, T-cell, Others). Each sub-network is a three-layer MLP with 1,024 hidden units and ReLU activations that takes the shared image feature as input and outputs a 512-dimensional representation specialized for classifying subtypes within its assigned superclass. For each image patch, the three sub-network outputs are combined as a weighted sum using the gating weights, producing a single aggregated feature vector per patch. This aggregation means that patches with strong B-cell FCM signals will be processed predominantly through the B-cell expert network, while patches from FCM-negative cases will draw on multiple experts weighted by image evidence.
After expert aggregation, each image patch in the bag has a single feature vector representing the weighted combination of expert outputs. These per-patch aggregated features are fed into an attention-based Multiple Instance Learning module. The attention network is a three-layer MLP with 128 hidden units and hyperbolic tangent activation that outputs a scalar attention weight for each patch. The weighted sum of all patch features, using normalized attention weights, produces a single bag-level feature vector. A final three-layer MLP classifier with 128 hidden units converts this bag-level feature into a six-dimensional class prediction through softmax activation, and the model is trained by minimizing cross-entropy loss between predictions and one-hot encoded ground truth labels.
Pre-training strategy: Effective training of the full model requires pre-trained initialization for both the shared feature extractor and the expert sub-networks, since the MoE architecture would otherwise not converge reliably to specialized experts. The pre-training proceeds in two stages. First, a standard six-class attention-based MIL model is trained on all 600 cases, producing a ResNet50 feature extractor capable of representing all six classes. Second, three separate two-class MIL models are trained, one per superclass, using only the cases from that superclass. These two-class models specialize the sub-networks for discriminating within each superclass (e.g., DLBCL vs. FL for the B-cell sub-network). The pre-trained feature extractor parameters are fixed during this second stage. The full multimodal MoE-MIL model is then initialized with these pre-trained components and fine-tuned end-to-end.
Training details: Input bags consist of 100 randomly selected 224 by 224-pixel patches per WSI, with the same 18-dimensional FCM vector replicated across all patches in the bag. Up to 50 bags are created per WSI per epoch, with no patch overlap within an epoch. Optimization uses SGD with momentum (0.9), learning rate 0.001 for all methods except the six-class pre-training step (0.01), with cosine annealing reducing the learning rate by a factor of 0.1 every 5 epochs over 10 epochs total. Sub-networks can optionally be re-trained during fine-tuning (referred to as the "Trained" variant) or kept frozen at their pre-trained values ("Frozen" variant). At test time, the class predictions from all 50 bags for a given case are averaged to produce the final case-level prediction.
Two types of attention in the model: The proposed model generates two distinct sets of attention weights that together enable rich visualization of the diagnostic reasoning. Gating weights (from the multimodal gating network) indicate which superclass each image patch is associated with, reflecting whether the patch shows features characteristic of B-cell, T-cell, or Other lymphoma subtypes. Attention weights (from the MIL attention module) indicate which patches are most important for the final classification within the selected superclass. When both weights are large for a given patch, that region is classified as a superclass-specific tumor region contributing to the final prediction.
Performance was evaluated using five-fold cross-validation, with 60% of cases used for training, 20% for validation, and 20% for testing in each fold, stratified to ensure all WSIs appeared in the test set at least once. The primary metrics are accuracy, macro-averaged precision, and macro-averaged AUC. Results are reported as mean and standard error across the five folds.
Baseline comparison: The study compares six methods. Method 1 is an optimistic oracle baseline that assumes perfect superclass knowledge and uses the correct two-class sub-network for testing, achieving 84.2% accuracy with AUC 0.980. This represents an upper bound for methods that use only sub-network classification with the correct superclass known in advance. Method 2, a simple six-class attention-based MIL using only images, achieves 70.2% accuracy and AUC 0.909. Methods 3 through 5 introduce FCM data in different ways. Method 3 (hierarchical classification: FCM-based superclass prediction followed by the appropriate two-class image model) achieves only 57-58% accuracy because FCM-based superclass classification using an MLP reaches only 83.1% accuracy, and errors at the superclass level propagate to the final prediction. Methods 4 and 5 incorporate MoE architecture: Method 4 uses FCM alone for gating (same weight for all patches), achieving 55-66% accuracy. Method 5 uses image patches alone for gating (patch-specific weights but no FCM), achieving 55-67% accuracy.
Proposed method performance: The proposed multimodal gating approach, which combines both image and FCM features per patch for gating, achieves 72.3% accuracy (AUC 0.924) in its best configuration (Trained sub-networks, T=0.8), compared to 70.2% for image-only MIL (Method 2) and 69.5% for the straightforward FCM-image combination baseline. The precision reaches 73.7% in the best configuration. The improvement over method 2 demonstrates that multimodal gating adds discriminative power beyond what images alone can provide, despite the unreliability of FCM data. The improvement over method 3 demonstrates that soft, multimodal gating is more robust than hard, FCM-only superclass routing when FCM signals are absent.
Confusion matrix analysis: The confusion matrix for the proposed method shows strong performance on B-cell subtypes: 74/100 DLBCL and 80/100 FL correctly classified, matching Method 3's performance on those classes. The multimodal gating particularly improves AITL classification, correctly identifying 67/100 AITL cases compared to 30/100 in Method 3. This is because AITL cases frequently lack T-cell FCM markers, and FCM-only gating misroutes these cases to the Others sub-network. The image features in the multimodal gate correctly redirect these cases back to the T-cell sub-network. The trade-off is some increased CHL misclassification as T-cell cases, a new confusion pattern introduced by the multimodal gating strategy.
A critical strength of the proposed architecture is its ability to generate biologically interpretable visualizations by overlaying computed weights onto the tissue slide. For visualization, gating and attention weights are computed for all image patches extracted from the entire WSI (not just the 100-patch bags used during training). Gating weights are displayed as heatmaps ranging from blue (weight near 0) to red (weight near 1), separately for each of the three sub-networks. The multiplication of gating weights by attention weights is then visualized as a second heatmap, where red regions are interpreted as class-specific tumor regions contributing to the final prediction.
ATLL case (T-cell with abnormal FCM): In an ATLL case where FCM data shows strong T-cell-related abnormal cell populations, the gating weights for sub-network 2 (T-cell expert) are high across almost the entire tissue region, reflecting the FCM signal's dominance. After multiplying by attention weights, the high-value regions concentrate in the interfollicular areas, correctly avoiding normal follicular structures. The model focuses on these interfollicular T-cell-rich regions for its final ATLL prediction, which is consistent with the known histomorphological pattern of ATLL.
CHL case (FCM-negative, correctly handled): A CHL case where FCM shows no abnormal cell populations is correctly classified by the model despite the absence of FCM evidence. The gating network assigns high weights to sub-network 3 (Others expert) based on image features, correctly routing the case to the expert trained on CHL and RL. Attention weights highlight regions of low cell-nucleus density, corresponding to areas where Hodgkin cells (characteristic large binucleated Reed-Sternberg cells) tend to appear. A high-magnification inset confirmed the presence of actual Hodgkin cells in these highlighted regions, providing pathologist-validated evidence that the model's attention correctly identifies diagnostically meaningful tissue areas.
AITL cases (T-cell without abnormal FCM): Two AITL cases with completely zero FCM vectors illustrate the multimodal gating advantage. FCM alone would force these cases to be classified as Others (since zero FCM indistinguishable from CHL and RL). However, with image features input to the gating network simultaneously, the model correctly emphasizes sub-network 2 outputs and classifies both cases as AITL. The visualization shows that the image features in these cases capture tissue-level characteristics, such as proliferating histiocytes in light-colored regions, that distinguish AITL from CHL even when FCM is uninformative. This case-level visualization was reviewed by an expert hematopathologist with over 15 years of experience diagnosing more than 10,000 lymphoma cases, who confirmed that the highlighted regions align with known diagnostic criteria.
Restricted class coverage: The current study evaluates only six lymphoma classes organized into three superclasses, whereas the WHO classification recognizes approximately 100 distinct lymphoma entities. The authors explicitly frame this as an initial proof-of-concept study. Scaling the MoE framework to cover clinically relevant additional subtypes such as mantle cell lymphoma and MALT lymphoma would require substantially more data and additional expert sub-networks. As the number of classes increases toward the full clinical complexity of lymphoma diagnosis, class imbalance becomes a significant challenge, since many rare subtypes have far fewer cases than the 100 per class used here.
Single-center dataset with no external validation: The entire dataset of 600 cases originates from a single institution (Kurume University), with FCM data provided by a single laboratory company. Tissue processing protocols, staining quality, WSI scanner calibration, and FCM panel composition all vary between institutions, and these variations are well-documented as sources of performance degradation when models are deployed outside their training domain. The study does not include external validation on data from a second institution. Domain adversarial learning, which has been applied successfully for multi-site lymphoma WSI classification in prior work, is identified as a relevant approach for addressing cross-site generalization.
FCM availability assumption: The model design assumes that FCM data will always be available at test time. In practice, FCM may not be performed for all lymphoma cases, particularly in resource-limited settings or when specimen quality is insufficient for flow cytometric analysis. The model does not have an explicit fallback pathway for cases where FCM input is absent or unreliable beyond the image-based compensation built into the gating network.
Interpretability caveats: While the dual-weight visualization provides rich spatial information, the correlation between visualized regions and ground-truth tumor areas has not been formally quantified through pixel-level pathologist annotation. The visualization results were reviewed qualitatively by a single expert pathologist on selected representative cases. A rigorous evaluation would require systematic annotation of tumor regions across a larger set of cases and statistical comparison of model attention maps against pathologist-drawn annotations, analogous to the evaluation methodology used in detection-oriented computational pathology studies.
Expanding class coverage: The most direct extension is to increase the number of target classes toward the full spectrum of clinically encountered lymphoma subtypes. The MoE architecture is well-suited to this expansion because additional expert sub-networks can be added for new superclasses, and the gating mechanism can be extended to route cases to the appropriate expert. The authors note that with more classes, the number of superclasses themselves may need to be reconsidered, since the current three-superclass organization was chosen to reflect the structure of FCM information rather than a comprehensive biological taxonomy.
Addressing class imbalance: Real-world lymphoma datasets have highly skewed class distributions, with common subtypes such as DLBCL and FL representing the majority of cases while rarer entities have far fewer specimens. The authors cite focal loss and class-balanced sampling as established techniques for handling class imbalance in computer vision tasks that could be adapted to the lymphoma classification setting. These approaches would be essential for extending the MoE-MIL framework to the full clinical case mix.
Domain adaptation for multi-site deployment: The paper identifies domain adversarial learning as a specific technical direction for enabling the model to generalize across institutions with different staining protocols and scanner hardware. Domain adversarial training adds a network component that penalizes the model for encoding site-specific information in its feature representations, forcing it to learn features that are discriminative for lymphoma classification rather than for distinguishing between institutions. This approach has been validated for the related task of multi-institution lymphoma WSI classification in prior work from the same research group.
Generalization beyond lymphoma: The MoE-MIL framework is not inherently specific to lymphoma. Any medical classification problem with a hierarchical class structure and multiple data modalities of different diagnostic utility could benefit from this approach. The gating mechanism could incorporate any auxiliary data that is informative for high-level category discrimination, not just FCM: examples from other cancer types include molecular subtype markers in breast cancer (ER, PR, HER2 status), PSA levels in prostate cancer, or radiological staging in lung cancer. This broader applicability positions the method as a general contribution to multimodal computational pathology rather than a lymphoma-specific tool.