{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-transcriptomics-foundation","title":"Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all","arxiv_id":"2410.13956","date":"2024-10-17","proceeding":null,"authors":["Ihab Bendidi","Shawn Whitfield","Kian Kenyon-Dean","Hanene Ben Yedder","Yassir El Mesbahi","Emmanuel Noutahi","Alisandra K. Denton"],"abstract":"Understanding the relationships among genes, compounds, and their interactions in living organisms remains limited due to technological constraints and the complexity of biological data. Deep learning has shown promise in exploring these relationships using various data types. However, transcriptomics, which provides detailed insights into cellular states, is still underused due to its high noise levels and limited data availability. Recent advancements in transcriptomics sequencing provide new opportunities to uncover valuable insights, especially with the rise of many new foundation models for transcriptomics, yet no benchmark has been made to robustly evaluate the effectiveness of these rising models for perturbation analysis. This article presents a novel biologically motivated evaluation framework and a hierarchy of perturbation analysis tasks for comparing the performance of pretrained foundation models to each other and to more classical techniques of learning from transcriptomics data. We compile diverse public datasets from different sequencing techniques and cell lines to assess models performance. Our approach identifies scVI and PCA to be far better suited models for understanding biological perturbations in comparison to existing foundation models, especially in their application in real-world scenarios.","url_abs":"https://arxiv.org/abs/2410.13956v2","url_pdf":"https://arxiv.org/pdf/2410.13956v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-transcriptomics-foundation","repo_url":"https://github.com/valence-labs/Tx-Evaluation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"pca","method_name":"PCA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.13956","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.13956"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/valence-labs/Tx-Evaluation","reach":null}],"summary":{"ran_fixture":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b5626eb6fa091d49","entry":"pert_signal_consistency_benchmark","repo":"valence-labs/Tx-Evaluation","repo_kind":"official","path":"biomodalities/eval/bmdb.py","file_url":"https://github.com/valence-labs/Tx-Evaluation/blob/HEAD/biomodalities/eval/bmdb.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5626eb6fa091d49"}},{"code_sha256_prefix":"df1a582fa4211886","entry":"pert_signal_consistency_metric","repo":"valence-labs/Tx-Evaluation","repo_kind":"official","path":"biomodalities/eval/bmdb.py","file_url":"https://github.com/valence-labs/Tx-Evaluation/blob/HEAD/biomodalities/eval/bmdb.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df1a582fa4211886"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}