{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/foundation-models-meet-imbalanced-single-cell","title":"Foundation Models Meet Imbalanced Single-Cell Data When Learning Cell Type Annotations","arxiv_id":null,"date":"2023-10-27","proceeding":"bioRxiv 2023 10","authors":["Abdel Rahman Alsabbagh","Alberto Maillo Ruiz de Infante","David Gomez-Cabrero","Narsis A. Kiani","Sumeer Ahmad Khan","Jesper N. Tegnér"],"abstract":"With the emergence of single-cell foundation models, an important question arises: how do these models perform when trained on datasets having an imbalance in cell type distribution due to rare cell types or biased sampling? We benchmark three foundation models, scGPT, scBERT, and Geneformer, using skewed single-cell cell-type distribution for cell-type annotation. While all models had reduced performance when challenged with rare cell types, scGPT and scBERT, performed better than Geneformer. Notably, in contrast to scGPT and scBERT, Geneformer uses ordinal positions of the tokenized genes rather than actual raw gene expression values. To mitigate the effect of a skewed distribution, we find that random oversampling, but not random undersampling, improved the performance for all three foundation models. Finally, scGPT, using FlashAttention, has the fastest computational speed, whereas scBERT is more memory-efficient. We conclude that tokenization and data representation are essential areas of research, and new strategies are needed to mitigate the effects of imbalanced learning in single-cell foundation models. Code and data for reproducibility are available at https://github.com/SabbaghCodes/ImbalancedLearningForSingleCellFoundationModels.","url_abs":"https://www.biorxiv.org/content/10.1101/2023.10.24.563625.abstract","url_pdf":"https://www.biorxiv.org/content/10.1101/2023.10.24.563625v1.full.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"foundation-models-meet-imbalanced-single-cell","repo_url":"https://github.com/SabbaghCodes/ImbalancedLearningForSingleCellFoundationModels","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"cell-entity-annotation","task_name":"Cell Entity Annotation"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"single-cell-modeling","task_name":"Single-cell modeling"},{"task_slug":"imbalanced-classification","task_name":"imbalanced classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}