Foundation Models for Single-Cell Biology: Architecture and Training at Scale

The idea of foundation models, large models pre-trained on broad data that can be adapted to many downstream tasks, has transformed natural language processing and computer vision. At Gimerny AI, we have spent two years building and deploying a foundation model for biology, specifically for single-cell and multi-omics data. GimernyGenome's core is a 1.2 billion parameter transformer that learns universal representations of biological states from 4.2 petabytes of sequencing data spanning 47 species, 312 tissue types, and 1,847 experimental conditions.

This post describes the architecture, training methodology, and downstream performance of our biological foundation model.

Why Foundation Models for Biology?

Traditional bioinformatics tools are task-specific. One tool performs differential expression analysis, another does cell-type annotation, a third performs trajectory inference. Each requires its own input format, its own assumptions, and its own limitations. Switching between tools is friction-laden, and combining results across tools requires ad-hoc integration.

A foundation model offers a different paradigm: learn a general-purpose representation of biological states, then adapt that representation to specific tasks with minimal fine-tuning. The same pre-trained model that powers cell-type annotation can also perform perturbation response prediction, disease state classification, and cross-species comparison, because all these tasks share the underlying biology.

The analogy to language models is instructive. GPT and BERT were not trained to do sentiment analysis or question answering specifically. They learned general representations of language that transfer to these tasks. Similarly, our biological foundation model is not trained for any specific biological task. It learns general representations of cellular states that transfer across tasks and contexts.

Architecture

Our model architecture is a modified transformer with several biology-specific adaptations. The input representation treats each gene's expression value as a token, with gene identity encoded via learned embeddings and expression level encoded via binning (we use 512 expression bins plus a zero-expression token). For a typical single-cell RNA-seq profile measuring 20,000 genes, this produces a sequence of 20,000 tokens.

Processing 20,000 tokens with standard self-attention is computationally prohibitive (quadratic complexity). We use a sparse attention pattern that combines local attention (each gene attends to its neighbors in a gene-gene interaction graph), global attention (a set of 256 "summary" tokens that attend to all genes and are attended by all genes), and pathway attention (genes within the same KEGG/Reactome pathway attend to each other).

This sparse attention pattern reduces the computational cost from O(n^2) to approximately O(n * sqrt(n)) while preserving the ability to capture long-range gene-gene dependencies through the summary tokens.

The model has 1.2 billion parameters distributed across 24 transformer layers, with 1024-dimensional hidden states and 16 attention heads. We use pre-normalization (LayerNorm before attention and FFN), rotary position embeddings adapted for our graph-based gene ordering, and SwiGLU activation functions.

Pre-Training Objectives

We use three complementary pre-training objectives. Masked gene modeling (MGM), analogous to masked language modeling, randomly masks 15% of genes' expression values and trains the model to predict the masked values from context. This teaches the model about gene co-expression patterns and regulatory relationships. Expression denoising (ED) adds synthetic noise to expression measurements (calibrated to match known technical noise distributions for 10x Chromium, Smart-seq2, and other platforms) and trains the model to recover the clean signal. This teaches the model about measurement noise and improves robustness to technical variation. Cross-modal prediction (CMP) receives one data modality (e.g., transcriptomics) as input and predicts another modality (e.g., surface protein levels measured by CITE-seq). This teaches the model about relationships between molecular layers and enables imputation of unmeasured modalities.

Training at Scale

Training a 1.2B parameter model on 4.2PB of data is a significant engineering challenge. We use 512 NVIDIA A100 GPUs for pre-training, with fully-sharded data parallel (FSDP) to distribute model parameters and optimizer states. Training runs for approximately 3 weeks (500,000 steps with a batch size of 2048 cells).

Data loading is a critical bottleneck. We pre-process all sequencing data into a tokenized format stored in Apache Parquet files on S3, with a streaming dataloader that prefetches and decodes data in parallel with GPU computation. The total pre-processed dataset is approximately 120TB after tokenization and compression.

We use a cosine learning rate schedule with 5,000 warmup steps, peaking at 3e-4, and train with bfloat16 mixed precision. Gradient clipping at norm 1.0 and a weight decay of 0.1 stabilize training.

Downstream Performance

We evaluate the pre-trained model on 12 downstream tasks spanning four categories. Cell-type annotation involves identifying cell types from expression profiles. On the Tabula Sapiens benchmark, fine-tuning our foundation model achieves 96.2% accuracy, compared to 91.7% for scANVI and 89.3% for a randomly initialized model of the same size. This demonstrates that pre-training captures biologically meaningful structure.

Perturbation response prediction involves predicting how cells respond to genetic or chemical perturbations. On the sci-Plex dataset (chemical perturbations) and Perturb-seq datasets (genetic perturbations), our model outperforms the previous state-of-the-art (GEARS) by 18% in mean squared error, suggesting that pre-trained representations encode regulatory logic that enables generalization to unseen perturbations.

Cross-species transfer involves training on human data and testing on mouse (or vice versa). Our model's pre-trained representations align homologous cell types across species without explicit species labels, achieving 89% accuracy on cross-species cell-type matching.

Disease state classification involves identifying diseased vs. healthy cells and characterizing disease-specific programs. On a held-out COVID-19 PBMC dataset, fine-tuned models achieve 94% accuracy in identifying infected cell states and correctly identify the top 20 differentially expressed genes in 18 of 20 cases.

What Pre-Training Does Not Solve

Pre-training is not magic. There are clear limitations. Context length limits mean that our model currently processes one cell at a time, missing cell-cell interactions and spatial context. Spatial transcriptomics data, where the positions of cells in tissue matter, requires architectural extensions we are actively developing. Rare cell types with fewer than 100 examples in the pre-training data are not well-represented in the learned embeddings. We mitigate this with few-shot learning techniques, but performance on truly rare cell types remains limited. Batch effects between laboratories and platforms are reduced but not eliminated by pre-training. We recommend running batch correction (e.g., Harmony or scVI) on the foundation model's embeddings for multi-site studies.

Open Questions

The field of biological foundation models is young, and several important questions remain open. How large should models be? We observe clear scaling laws (performance improves log-linearly with model size up to 1.2B parameters), but we do not yet know where the scaling curve plateaus. What data is most valuable for pre-training? Our ablation studies suggest that perturbation data (Perturb-seq, CRISPR screens) contributes disproportionately to downstream performance, even though it represents less than 5% of the pre-training data. How should we handle multi-modal data? Our current cross-modal prediction objective is a start, but more sophisticated multi-modal architectures (joint encoders, cross-attention between modalities) may capture richer cross-modal relationships.

Foundation models for biology are still in their early days, but the direction is clear: general-purpose biological representations, pre-trained at scale, will become the standard starting point for computational biology just as pre-trained language models have become the standard for NLP.