Every cell in a tumor does not behave the same way. Cells at the invasive edge of a tumor behave differently from those deep in the core. Immune cells respond to signals from their immediate neighbors. Understanding cancer requires knowing not just what genes are active, but where in the tissue they are active and which cells are nearby. Traditional molecular biology destroys this spatial information by grinding tissue into a suspension of isolated cells before measuring gene expression.
Spatial transcriptomics was developed to preserve location. Rather than separating cells, these technologies measure gene activity at defined positions within an intact tissue slice, creating a map that links molecular data to tissue architecture. Platforms like 10X Visium produce a grid of spots across the tissue, each associated with the expression levels of thousands of genes and its precise spatial coordinates.
At the same time, advances in imaging AI have made it possible to extract detailed information from histological images, the stained tissue photographs that pathologists read under the microscope. Deep learning models can identify cell types, detect tumor boundaries, and quantify structural features from these images with remarkable precision.
These two data streams, molecular gene expression maps and AI-analyzed tissue images, contain overlapping but not identical information about tissue biology. This review introduces a systematic framework for understanding how to combine them, clarifying when image morphology can substitute for gene expression data and when it adds genuinely new information that gene expression alone cannot provide.
The core insight of this review is that morphological features from tissue images can relate to gene expression in two fundamentally different ways, and confusing the two leads to flawed analyses. The authors introduce a framework organized around two key dimensions: how relevant the morphological features are for the biological task at hand, and how much information they share with gene expression.
Translation describes the scenario where tissue morphology captures essentially the same biological signal as gene expression. In this case, the image can be used to predict or substitute for gene expression data. This is valuable because obtaining histological images is far cheaper and faster than running a spatial transcriptomics experiment. Translation is the basis for tools that generate gene expression predictions from standard clinical tissue slides.
Integration describes the opposite scenario: morphological features that contain relevant biological information not captured by gene expression. Because gene expression and tissue appearance change on different timescales, a tissue's visual structure may reflect biological history that is no longer visible in current gene activity patterns. Integration adds this complementary information to enrich analysis rather than replace the molecular data.
Two additional scenarios define what should not be combined with gene expression: noise, where morphological features are irrelevant and uncorrelated with gene expression; and overestimation, where features correlate with gene expression but only for genes that have no clinical relevance, such as housekeeping genes active in all cells. Using features from these quadrants inflates performance metrics without providing genuine biological insight.
The primary application of morphology translation is gene expression prediction: training a deep learning model to take an image patch as input and output the expected gene expression levels at that location. If successful, this would allow researchers to infer molecular information from the millions of archived tissue slides stained with standard protocols, without running expensive spatial transcriptomics experiments.
The field began with ST-Net, which used a DenseNet architecture to predict breast cancer gene expression from H&E images, showing moderate but meaningful correlations with known cancer biomarkers. Subsequent methods introduced increasingly sophisticated architectures. Vision Transformers (ViTs) replaced convolutional networks for their ability to capture long-range relationships across the entire image, while graph neural networks (GNNs) modeled the spatial relationships between neighboring tissue spots, mimicking the way gene expression is influenced by cellular neighborhood.
A critical design choice is which genes to train the model to predict. Spatial transcriptomics measures nearly 20,000 genes, but most lack meaningful spatial patterns or are expressed at levels too low to predict reliably from images. Researchers select either the most highly expressed genes, the most spatially variable genes (those whose activity changes most dramatically between tissue regions), or genes chosen based on prior biological knowledge. This gene selection step determines whether a translation model falls in the useful quadrant of the framework or inflates its performance by predicting biologically irrelevant genes.
More recent methods have pushed toward super-resolution gene expression prediction, attempting to infer gene activity at the level of individual pixels or cells rather than the larger spots used by standard sequencing-based platforms. XFuse and iStar demonstrated that H&E images contain enough structural information to predict gene expression at finer spatial scales than the original molecular measurements, validated by comparison with higher-resolution imaging-based spatial transcriptomics platforms.
Spatial domain identification is the primary goal of morphology integration. The aim is to divide a tissue slice into meaningful regions, such as tumor nests, stromal zones, immune infiltrates, or necrotic areas, by combining gene expression patterns with visual structural features that gene expression alone does not fully capture.
Early integration methods like SpaCell and stLearn used CNNs pre-trained on ImageNet to extract image features, then combined them with gene expression for clustering. SpaGCN built a spatial graph where the distances between nodes incorporated both gene expression similarity and image color intensity values. More recent approaches use contrastive learning, training the model to bring together image and gene expression features from the same tissue spot while pushing apart features from different spots, forcing the two data streams into a shared representation.
The fundamental challenge for integration is that features extracted from networks pre-trained on natural photographs may not be relevant to tissue biology. A network trained to distinguish cats from dogs learns visual features optimized for those categories, not for distinguishing cancer stroma from healthy epithelium. This mismatch may explain why current integration methods have not demonstrated consistent benefits over using gene expression data alone.
A key biological insight motivating integration is that gene expression and morphology operate on different timescales. A cell can change its gene activity rapidly in response to signals, but the visible structural consequences, such as changes in nuclear size, cell shape, or tissue architecture, take longer to manifest. Morphology therefore holds a partial record of the cell's history that current gene expression does not capture. This temporal decoupling provides the theoretical basis for integration to add genuinely new information.
Evaluating whether combining morphology with gene expression actually improves analysis is more complicated than it appears. For translation tasks, the standard metric is Pearson's correlation coefficient (PCC) between predicted and measured gene expression. However, achieving high PCC on genes that are uniformly expressed across all tissue regions, called housekeeping genes, provides no diagnostic or clinical value. A model can look good on paper while predicting nothing biologically meaningful.
For integration tasks, the standard metric is the adjusted Rand index (ARI), which measures how well the tissue domains identified by the combined model match expert annotations. A complementary approach uses Moran's I, a measure of spatial autocorrelation, to assess whether the genes defining identified domains show consistent spatial patterns across the tissue. Higher Moran's I indicates that the identified domain boundaries correspond to real biological transitions in gene activity.
A structural weakness of much published evaluation is that high performance may reflect overfitting to a specific dataset rather than generalizable biological insight. Many studies use leave-one-out cross-validation within a single cohort, which tests whether the model memorizes patterns in one dataset but does not test whether it generalizes to patients from different hospitals or different imaging platforms. True clinical utility requires validation on completely independent datasets.
The field lacks widely accepted benchmarks that test all methods on the same datasets with the same metrics, making it difficult to determine whether newer methods genuinely improve on older ones. Two benchmarking studies that attempted head-to-head comparisons found inconsistent performance across different datasets and metrics, underscoring the need for more rigorous and standardized evaluation standards before these tools are applied to clinical decision-making.
Although this review covers spatial transcriptomics broadly, it has direct relevance to prostate cancer research. The authors note that networks trained specifically for cancer classification tasks have been found to already contain features correlating with genes associated with prostate cancer. This suggests that AI models trained on prostate tissue slides implicitly learn some of the molecular biology of the disease, even when not explicitly trained on genomic data.
Building on this observation, the authors developed MHAST, a framework that uses self-supervised morphological features to guide the spatial assignment of cells from single-cell deconvolution, improving the accuracy of cell-level resolution in spatial transcriptomics data. This type of approach could be used to map the spatial distribution of cancer cell subtypes, immune cells, and stromal populations within prostate tumors at single-cell resolution.
Foundation models for histopathology, large AI models pre-trained on millions of tissue slides using self-supervised learning, represent the most promising near-term advance for the field. Models such as UNI, GigaPath, and Virchow have demonstrated superior feature representations compared to networks trained on natural images. However, integrating these models into spatial transcriptomics analysis remains largely unexplored, and realizing their potential requires specifically training or fine-tuning them to extract features that are both clinically relevant and complementary to gene expression.
Gene expression prediction from H&E images currently shows only moderate correlation values with true gene expression. The review is candid that performance is far from clinically transferable at this stage. Achieving reliable prediction for clinically important genes, not just the most easily predicted ones, will require specialized models trained explicitly to target genes with demonstrated relevance to diagnosis, prognosis, or treatment response.
The combination of spatial transcriptomics and tissue morphology is a rapidly developing field that promises a more complete picture of cancer biology than either data source can provide alone. The translation-integration framework introduced in this review provides a principled way to think about when combining the two data types adds genuine value and when it risks inflating performance through biologically irrelevant correlations.
The most immediate practical insight is that spatial domain identification has not yet meaningfully benefited from adding morphological features, likely because current feature extractors are not sufficiently tissue-specific. Future work must prioritize developing descriptors that are both highly relevant to the biological task and genuinely complementary to what gene expression already captures, avoiding the inclusion of redundant or noisy features.
The temporal dimension of the relationship between gene expression and morphology remains largely unexplored. Because tissue structure reflects biological history while gene expression captures the current molecular state, morphology may contain information about past states or predict future changes, offering a unique window into tumor progression dynamics that no other data source provides.
As spatial multi-omics technologies continue to expand beyond gene expression to include the epigenome, proteome, and metabolome, the framework can be extended to guide integration of multiple biological data layers with morphology simultaneously. The ultimate goal is a unified model of tissue biology that can be read from a standard tissue slide at a cost that brings molecular insight within reach for every cancer patient.