Automated breast cancer diagnosis from histopathological microscopy images typically involves classifying tissue into benign or malignant categories, and ideally into specific tumor subtypes. While Vision Transformers (ViTs) have demonstrated excellent classification accuracy, they require significant computational resources due to their multi-head self-attention mechanism, which scales quadratically with the number of image patches. This limits their practical deployment in clinical environments without high-end GPUs.
Convolutional neural networks (CNNs) are faster and more parameter-efficient, but are limited in their ability to capture long-range spatial dependencies between tissue regions. A tumor's malignancy may be determined not only by individual cell characteristics but by the spatial relationships between cells across the tissue sample, which CNNs cannot effectively model with their limited local receptive fields.
A key challenge specific to histopathology is that high-resolution tissue images must be divided into patches for processing, but using all patches increases computational cost dramatically. Much of the patch content may be uninformative (background tissue, processing artifacts), yet standard approaches process all patches equally. Selectively identifying and focusing on the most diagnostically relevant patches could simultaneously improve accuracy and reduce computational burden.
This study introduces TokenMixer, a hybrid architecture that combines tokenization-based patch selection (using CNN-based attention maps) with ViT encoder processing. By dynamically learning which image patches are most informative and processing only those through transformer layers, TokenMixer achieves near-ViT accuracy with near-CNN computational efficiency -- a critical advantage for real-world clinical deployment.
The TokenLearner component addresses the patch redundancy problem by learning to dynamically select informative spatial regions from the input image. Rather than treating all image patches equally, it generates spatial attention maps that assign importance weights to different image regions, multiplying these weights element-wise with the input to produce a reduced set of representative tokens.
For each image, TokenLearner generates S tokens from the input frame, where each token represents a spatially weighted summary of the image. A learned tokenizer function maps each spatial region to a token vector through spatial global average pooling, progressively compressing the high-dimensional image representation into a compact, information-dense token sequence. This adaptive tokenization allows the model to concentrate computational resources on the regions most relevant for distinguishing benign from malignant tissue.
The key advantage of adaptive tokenization over fixed patch selection is that the token positions are not predetermined by the patch grid but are instead learned from the data. The model discovers which tissue regions -- such as areas with high cell density, irregular nuclear morphology, or disrupted glandular architecture -- are most discriminative for each specific image, enabling more efficient feature extraction than applying uniform attention across all patches.
By reducing the total number of tokens fed to subsequent transformer layers, TokenLearner significantly reduces the computational cost of the multi-head self-attention operation, which scales quadratically with sequence length. Processing 16 informative tokens rather than 256 full-image patches reduces attention computation by a factor of 256, enabling much faster training while preserving the transformer's ability to model spatial relationships between the selected informative regions.
ConvMixer processes image patches using two types of convolutions applied directly at the patch level: depthwise convolution that mixes spatial information within each channel, and pointwise (1x1) convolution that mixes channel information across the feature map. By maintaining consistent patch resolution throughout all layers and using large kernel sizes, ConvMixer achieves a broad receptive field without requiring explicit attention mechanisms.
TokenMixer combines the strengths of TokenLearner and ConvMixer. First, depth-wise convolution layers process input patches to generate attention weight maps, which are used to select and tokenize the most informative patches. ConvMixer layers then apply channel-wise mixing to the selected patches, extracting and refining local features. Finally, the reduced set of processed tokens is passed to a Transformer encoder that captures long-range dependencies between the selected informative regions.
The architecture processes 224x224 pixel input images using 14x14 pixel patches, resulting in 256 initial patches. After TokenMixer's adaptive selection, this is reduced to a much smaller set before the transformer encoder. The transformer encoder employs multi-head self-attention across these selected patches, followed by layer normalization and global average pooling, before a final MLP classification head using sigmoid activation for binary classification or softmax for multi-class output.
A critical design principle is that TokenMixer is trained from scratch without relying on ImageNet pretraining or transfer learning. This distinguishes it from many competing approaches and demonstrates that the architecture itself, rather than pre-learned knowledge from natural images, is responsible for the performance gains. Training from scratch on limited medical datasets is significantly harder, making the competitive results more meaningful.
The study used the publicly available BreakHis dataset containing 7,925 microscopic biopsy images from 82 patients, captured at four magnification levels (40x, 100x, 200x, 400x). The dataset includes 2,496 benign and 5,429 malignant images, with the 2.2:1 class imbalance addressed through data augmentation during training.
For binary classification, images were divided into benign versus malignant categories. The more demanding multi-class classification task required distinguishing eight distinct tumor subtypes: four benign (adenocarcinoma, fibroadenoma, phyllodes tumor, tubular adenoma) and four malignant (ductal carcinoma, lobular carcinoma, mucinous carcinoma, papillary carcinoma). The eight-class task is substantially harder due to smaller per-class sample sizes and subtle inter-class differences.
Different magnification levels reveal different diagnostic features. At 40x, overall tissue architecture and glandular organization are visible. At higher magnifications (200x, 400x), individual cell morphology, nuclear features, and mitotic figures become apparent. Each magnification was treated as a separate dataset, and models were evaluated independently at each level to identify which magnification provides the most diagnostic information for different classification approaches.
Data augmentation was applied to address class imbalance and limited dataset size: rotation range of 5 degrees, width and height shifts of 10%, and zoom range of 0.1%. These relatively conservative augmentation parameters preserve the diagnostic histopathological features while introducing sufficient variability to improve generalization. Three-fold cross-validation was used for the multi-classification task to ensure reliable performance estimates despite limited data.
The TokenMixer model achieved 97.02% accuracy for binary classification at 200x magnification, with precision of 96.33%, sensitivity of 96.74%, specificity of 96.74%, AUC of 99.70%, and F1 score of 96.53%. Critically, it achieved this performance in just 391.71 seconds of total training time, compared to 2,729.14 seconds for ConvMixer and 639.66 seconds for the standard ViT.
The speed advantage is striking: TokenMixer trained nearly 7 times faster than ConvMixer and 1.6 times faster than the standard ViT while maintaining competitive accuracy. The TokenLearner-alone model was similarly fast (391.29 seconds) but achieved lower accuracy (95.53%), confirming that the additional ConvMixer feature refinement in TokenMixer provides meaningful accuracy gains without significant time cost.
ConvMixer achieved the highest binary accuracy at 98.60% at 100x magnification -- slightly above TokenMixer -- but at 7x the training cost. The standard ViT reached 96.53% accuracy at 200x magnification but required 36.4 million parameters compared to TokenMixer's 1.4 million -- a 26-fold parameter reduction. This massive parameter efficiency makes TokenMixer substantially more practical for deployment on limited hardware.
All four models showed their best binary classification performance at either 100x or 200x magnification, suggesting that these magnification levels provide the optimal balance between tissue architecture visibility and cellular detail for distinguishing benign from malignant tumors. At 100x, both glandular organization and nuclear morphology are visible, likely providing the richest diagnostic information for binary classification.
For the eight-class tumor subtype classification task, TokenMixer achieved 93.29% accuracy at 40x magnification with a training time of 1,173.56 seconds, compared to 6,436.47 seconds for ConvMixer (94.71% accuracy) and 1,730.19 seconds for the standard ViT (93.67% accuracy). TokenMixer again achieved the best speed-accuracy balance, training 5.5x faster than ConvMixer with only 1.42 percentage points lower accuracy.
Interestingly, all four models achieved their best multi-class performance at 40x magnification, the lowest magnification level. This differs from binary classification where 100x or 200x was optimal. The authors suggest that at 40x, tissue architecture patterns that distinguish between different cancer subtypes (such as glandular versus lobular organization, or the presence of mucin pools in mucinous carcinoma) are most visible, while higher magnifications reveal cellular details that may be less discriminative between subtypes.
The eight-class task proved substantially harder than binary classification across all models, with accuracy dropping by approximately 3.5-4.5 percentage points. This gap reflects the challenge of limited per-class samples in the BreakHis dataset: with only approximately 100-200 images per subtype per magnification level, the models have limited examples to learn the subtle distinguishing features between, for example, ductal carcinoma and lobular carcinoma.
The authors note that the scarcity of high-magnification samples particularly limited multi-classification performance. At 200x and 400x magnifications, some tumor subtypes had as few as 108 training images -- an extremely small number for distinguishing among eight classes. This underscores that dataset size may be the primary bottleneck for multi-class histopathology AI, rather than architectural limitations.
A comprehensive comparison table shows TokenMixer achieving competitive results against 11 prior studies on the BreakHis dataset. Several prior methods reported higher binary classification accuracy (e.g., CNN-LSTM at 99.75%, Swin Transformer ensemble at 99.60%, DenseNet201 at 99.00%), but these models typically used pretrained ImageNet backbones, required substantially more parameters, and had significantly longer training times.
The study makes an important methodological contribution by demonstrating that training from scratch on limited medical data is feasible with TokenMixer's approach, achieving 97.02% binary accuracy -- comparable to many transfer learning approaches. This is significant because pretrained models may transfer dataset-specific biases from ImageNet (natural photographs) to histopathology, while a scratch-trained architecture must learn relevant features directly from tissue images.
Among ViT variants tested by Sriwastawa and Arul Jothi, MaxViT achieved 92.12% binary accuracy in 8.83 hours of training. TokenMixer achieved 97.02% in under 0.11 hours -- approximately 80 times faster while achieving 4.9 percentage points higher accuracy. This comparison highlights how the adaptive tokenization strategy fundamentally changes the efficiency equation for transformer-based histopathology analysis.
The study also acknowledges that pure CNN approaches like ConvMixer achieved higher binary accuracy (98.60%), suggesting that for straightforward binary classification the complexity of transformer attention may not be necessary. However, for the more clinically relevant multi-class subtype classification, TokenMixer's transformer component likely provides benefits in capturing the long-range spatial relationships between tissue regions that distinguish between subtle cancer subtypes.
TokenMixer demonstrates that hybrid CNN-Transformer architectures using adaptive tokenization can achieve near-state-of-the-art breast cancer histopathology classification with dramatically reduced computational requirements. The model achieved 97.02% binary and 93.29% multi-class accuracy while training 7x faster than ConvMixer and using 26x fewer parameters than standard ViT, establishing a compelling case for resource-efficient hybrid architectures in medical imaging.
The finding that all models peaked at 40x for multi-classification while performing best at 100-200x for binary classification has practical diagnostic implications. Clinical workflows could potentially use different magnification levels for different diagnostic questions: tissue architecture examination at 40x for subtype characterization, and higher magnifications for confirming benign-versus-malignant distinction -- mirroring expert pathologist practice.
The primary limitation is dataset size for multi-class classification. With only approximately 100-200 samples per subtype at higher magnifications, models cannot fully learn the feature space for reliable subtype classification. The authors recommend future investigation of hierarchical classification frameworks that first separate benign from malignant before distinguishing subtypes within each category, which could better utilize limited labeled data.
Future directions include extending TokenMixer to whole-slide image (WSI) analysis, which involves working with gigapixel images that contain entire tissue sections rather than pre-extracted patches. The adaptive tokenization approach is particularly promising for WSI analysis where patch selection efficiency is critical, and integration with vision-language models could enable natural language pathology report generation from classified tissue images.