Deeper evaluation of a single-cell foundation model

Large-scale foundation models, which are pre-trained on massive, unlabelled datasets and subsequently fine-tuned on specific tasks, have recently achieved unparalleled success on a wide array of applications, including in healthcare and biology (opens in new tab). The success of these models has showcased the power of leveraging generalizable features and contextual understanding to improve a model’s performance. Single-cell bidirectional encoder representations from transformers (scBERT) by Yang et al.^{7 (opens in new tab)} is one of several recently developed foundation models to learn representations of single-cell RNA-sequencing data. Yang et al. pre-trained their model on 1.12 million cells to impute masked gene-expression values and characterize the performance of their model on a fine-tuning task to annotate cell types. We reproduce their results, and provide additional baselines and ablation studies (that is, remove components of the model’s architecture or training process) to develop a deeper understanding of their results and the potential benefits and limitations of single-cell foundation models.