Vision Transformers (ViTs) are a class of deep learning models originally inspired by the transformer architecture developed for natural language processing. Rather than analyzing an image as a continuous pixel grid, ViTs divide images into a series of fixed-size patches and treat each patch as a discrete token, much like a word in a sentence.
The core innovation of ViTs is the self-attention mechanism, which allows the model to dynamically weigh the importance of each image patch relative to every other patch. This enables the network to capture long-range spatial relationships across an entire image, something that convolutional neural networks (CNNs) struggle with due to their reliance on fixed local receptive fields.
Traditional CNNs process images by applying learned filters to local regions and progressively pooling features. While effective, this approach can miss global context. ViTs overcome this limitation by computing attention across all patches simultaneously, making them particularly powerful for tasks where global context matters, such as identifying subtle patterns across an entire medical scan.
This review surveys ViT applications across 14 medical domains including breast cancer, skin lesions, brain tumors, lung diseases, retinal disorders, and COVID-19, systematically comparing architectures, datasets, and performance outcomes to provide a comprehensive reference for researchers and clinicians.
A ViT begins with patch embedding, where an input image is divided into non-overlapping square patches. Each patch is flattened and projected into a high-dimensional embedding space through a learnable linear transformation. The number of patches becomes the sequence length, analogous to words in a sentence passed to a language model.
Positional encoding is then added to each patch embedding to preserve spatial information, since transformers have no built-in sense of order or position. Without this step, the model could not distinguish a patch in the upper-left corner from one in the lower-right corner.
The transformer encoder processes the sequence through multiple layers, each containing two sub-components: a multi-head self-attention (MSA) layer and a feed-forward network (FFN). Self-attention computes query, key, and value matrices for each patch and uses their relationships to generate context-aware representations. Multiple attention heads allow the model to attend to different aspects of the image simultaneously.
A special learnable class token is prepended to the patch sequence. After passing through all transformer layers, this token aggregates global image information and is fed into a final classification layer with a softmax activation to produce class probabilities. This design allows a single forward pass to classify the entire image.
The Data-Efficient Image Transformer (DeiT) addresses the original ViT's need for massive training datasets by introducing a knowledge distillation framework. A larger teacher model guides a smaller student model through a dedicated distillation token, enabling competitive performance even with limited labeled data. DeiT also uses aggressive data augmentation techniques including Mixup, CutMix, and random erasing.
The Swin Transformer reduces the quadratic computational cost of global self-attention by computing attention only within local windows of fixed size. A shifted window mechanism moves these windows between alternating layers to allow information exchange between neighboring regions, achieving linear complexity relative to image size while preserving both local and global context.
ConViT blends convolutional and transformer approaches through a Gated Positional Self-Attention (GPSA) mechanism. Early layers mimic convolutional filters by enforcing locality constraints, then gradually relax these constraints during training, allowing the network to escape local patterns and capture global context. This hybrid approach inherits the sample efficiency of CNNs while retaining transformer flexibility.
LeViT prioritizes inference speed by replacing standard patch embeddings with a series of 3x3 convolutional layers and a pyramid pooling structure that progressively reduces spatial resolution. It uses learned spatial attention biases instead of traditional positional embeddings, and replaces GELU activation with Hardswish for computational efficiency, making it suitable for real-time clinical applications.
Multiple research groups have applied ViTs to breast cancer classification across different imaging modalities. Using ultrasound images, Gheflati and Rivaz demonstrated that pre-trained ViT models achieved 86.7% accuracy on the BUSI dataset, while Wang et al. developed the Attention Transformer-ViT (ATS-ViT) model combining supervised and semi-supervised training, reaching 98.12% accuracy on the same data.
For histopathology images, Tummala et al. applied Swin Transformer ensembles to the BreakHis dataset of 7,909 microscopic tissue images, achieving 96% accuracy for eight-class subtype classification and 99.6% for binary malignancy classification. Abimouloud et al. proposed a hybrid TokenMixer architecture that reduced computational cost while achieving 97.02% binary and 93.29% multi-class accuracy on BreakHis.
For mammography, Chen et al. developed the Multi-View Vision Transformer (MVT), which processes images from multiple views simultaneously using both local and global transformer blocks. This approach achieved an AUC of 0.818 on 3,796 mammograms from 949 patients, outperforming multi-view CNN baselines that achieved AUC values of 0.724 to 0.784.
He et al. enhanced ViTs with Discrete Wavelet Transform (DWT) input features for breast ultrasound, extracting frequency-domain characteristics that improve recognition of subtle tissue patterns. The wavelet-enhanced model achieved AUC scores of 0.984 and 0.968 on two independent datasets, demonstrating that combining signal processing techniques with transformers can further boost classification performance.
In skin lesion classification, transformer-based models have demonstrated strong performance on the HAM10000 dataset of 10,015 dermoscopic images. Sarker and Moreno-Garcia's bidirectional encoder model achieved 90.22% accuracy, while Himel et al. combined the Segment Anything Model (SAM) for lesion segmentation with multiple ViT classifiers, with the ViT-Google model reaching 96.15% accuracy and a very low false-negative rate for malignant lesions.
Xin et al. developed SkinTrans, enhancing ViTs with multi-scale overlapping sliding windows and contrastive learning to maximize feature differences between cancer subtypes. Their model achieved 94.3% accuracy on HAM10000 and 94.1% on an independent clinical dataset. Zhou et al. further improved performance by incorporating both image and textual clinical information through a Mutual Attention Transformer (MAT) that fuses data from both modalities.
For lung cancer detection, the Swin Transformer has been applied to CT scan classification, achieving 82.3% classification accuracy with over 95% accuracy on segmentation tasks. Imran et al. developed a CNN-ViT hybrid for non-small cell lung cancer (NSCLC) histopathology that classified adenocarcinoma and squamous cell carcinoma with 98.8% accuracy and an inference time of just 1.816 milliseconds, suitable for real-time pathology workflows.
For cytopathology, Chen et al. applied the Swin Transformer to classify individual lung cells from bronchial wash samples, achieving 96.16% accuracy in distinguishing cancerous from normal cells among 2,473 labeled cells from 10 patients. This approach could aid pathologists by pre-screening cellular samples and flagging suspicious cells for review.
In MRI brain tumor classification, fine-tuned ViT models have achieved near-perfect accuracy. Reddy et al. evaluated four ViT configurations on 7,023 MRI images across four tumor types (glioma, meningioma, pituitary, and no tumor), with the best model achieving 98.70% accuracy and outperforming CNN baselines such as ResNet-50 (96.5%) and EfficientNet-B0 (95.1%). Zhang et al. developed AugTransU-Net for tumor segmentation, achieving 89.7% Dice similarity for whole-tumor delineation.
For retinal disease classification using optical coherence tomography (OCT) images, ViTs have achieved exceptional performance. Jiang et al. applied a six-layer transformer encoder to classify retinal diseases including diabetic macular edema and AMD, achieving 99.69% classification accuracy. He et al.'s Swin-Poly Transformer reached 99.80% accuracy and an AUC of 99.99% on the OCT2017 dataset, setting new state-of-the-art benchmarks.
Wang et al. developed a retinal ViT for multi-label classification of retinal fundus images, demonstrating that a pure transformer architecture (without convolutional components) can match or exceed CNN performance. Their model achieved an AUC of 0.938 on the ODIR-2019 dataset covering diabetic retinopathy, glaucoma, myopia, and cataract.
For COVID-19 CT classification, combined CNN-transformer approaches proved effective. Mehboob et al.'s pure ViT model achieved 99.7% accuracy on the HUST-19 dataset by optimizing patch size, number of transformer layers, and training augmentation. Fan et al. used a bidirectional feature fusion of transformer global features and CNN local features, demonstrating that hybrid architectures can outperform either approach alone.
This comprehensive review systematically surveyed peer-reviewed articles applying vision transformers to medical image classification. The authors established explicit inclusion and exclusion criteria, prioritizing studies that reported quantitative performance metrics, used publicly available or clinically relevant datasets, and demonstrated methodological innovations beyond simple model application.
Studies were organized by medical application domain rather than by technical architecture, providing a disease-centric perspective that allows clinicians and researchers in a specific field to rapidly identify relevant work. This organizational approach distinguishes the review from previous surveys that focused on general architectural discussions.
For each study, the authors analyzed data preprocessing methodologies, model architecture choices, transfer learning approaches, interpretability methods, and identified limitations. Performance comparisons were made within each disease domain to identify which ViT variants and training strategies consistently achieve the highest accuracy.
The 14 medical domains covered include breast cancer, skin lesions, MRI brain tumors, lung diseases, retinal analysis, COVID-19, heart disease, colon cancer, brain disorders, diabetic retinopathy, skin diseases, kidney disease, lymph node disease, and bone analysis, making this one of the most comprehensive ViT medical imaging reviews published to date.
The fundamental architectural difference between ViTs and CNNs lies in their receptive field. CNNs use fixed-size convolutional kernels that capture local patterns, requiring many layers to achieve a large receptive field. ViTs, through self-attention, can relate any two patches in a single layer regardless of distance, making them naturally suited for detecting spatially distributed features such as tumor boundary irregularities or diffuse tissue changes.
A key practical limitation of original ViTs is their requirement for large training datasets. CNNs benefit from strong inductive biases (local connectivity, translation equivariance) that help them generalize from smaller samples. ViTs lack these biases and typically require pre-training on large datasets like ImageNet before fine-tuning on medical images, which are often scarce and expensive to label.
From a computational complexity standpoint, standard ViTs have quadratic scaling with respect to the number of patches, meaning doubling image resolution quadruples memory and computation. Swin Transformer's windowed attention achieves linear complexity, making high-resolution medical image analysis more feasible. Hybrid models like ConViT and LeViT further reduce computational requirements while maintaining accuracy.
Model interpretability remains a shared challenge for both CNNs and ViTs in medical settings. Attention maps from ViTs can highlight which image regions influenced a classification decision, but these do not always align with clinically meaningful features. Methods like Score-CAM, Grad-CAM, and Focused Attention have been developed to produce more interpretable explanations, which are essential for clinical acceptance of AI-driven diagnostic tools.
The consistently high accuracy of ViT-based models across diverse medical imaging tasks suggests substantial potential for clinical decision support. Models achieving over 96% accuracy in breast cancer histopathology classification could serve as a pre-screening tool to prioritize pathologist review, reducing workload and potentially decreasing diagnostic turnaround time.
For breast cancer specifically, ViTs demonstrate the ability to work across all major imaging modalities: ultrasound, mammography, and histopathology. This cross-modality capability means that a single architectural framework could potentially be adapted to support radiology, pathology, and screening workflows within a single institution.
A recurring theme across all reviewed studies is the need for larger and more diverse datasets to improve model generalizability. Most studies used relatively small single-institution datasets, raising concerns about whether models trained on one population would perform equally well on patients from different demographics, imaging equipment, or clinical protocols.
Real-time inference requirements vary by clinical setting. Radiology workflows may tolerate longer processing times for complex analyses, while intraoperative or point-of-care applications demand millisecond response times. LeViT and CCT variants designed for speed, combined with model distillation, point toward a path for deploying ViTs in time-critical clinical environments.
This review demonstrates that vision transformers consistently match or outperform CNNs across the majority of medical image classification tasks studied, with particularly strong results in breast cancer histopathology, retinal OCT classification, and brain tumor MRI analysis. The adoption of pre-trained ViT backbones combined with domain-specific fine-tuning has emerged as the most effective training strategy.
Key future directions include development of multi-modal ViT architectures that can simultaneously process imaging data alongside clinical text, genomic profiles, and laboratory values. Several studies in this review that incorporated additional data modalities achieved measurable performance gains, suggesting that models integrating diverse data streams may surpass single-modality approaches.
Improving model interpretability remains critical for clinical translation. Future work should focus on developing attention visualization methods that align with established clinical features and pathological criteria, enabling clinicians to understand and trust model predictions rather than treating them as opaque outputs.
Finally, standardized benchmarking on shared, multi-institutional datasets is needed to enable meaningful comparison across studies. The current landscape of fragmented datasets and inconsistent evaluation metrics makes it difficult to determine which ViT variants are truly superior for specific clinical tasks, and collaborative data-sharing initiatives could accelerate progress in this field.