{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/perturbench-benchmarking-machine-learning","title":"PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis","arxiv_id":"2408.10609","date":"2024-08-20","proceeding":null,"authors":["Yan Wu","Esther Wershof","Sebastian M Schmon","Marcel Nassar","Błażej Osiński","Ridvan Eksi","Zichao Yan","Rory Stark","Kun Zhang","Thore Graepel"],"abstract":"We introduce a comprehensive framework for perturbation response modeling in single cells, aimed at standardizing benchmarking in this rapidly evolving field. Our approach includes a modular and user-friendly model development and evaluation platform, a collection of diverse perturbational datasets, and a set of metrics designed to fairly compare models and dissect their performance nuances. Through extensive evaluation of both published and baseline models across diverse datasets, we highlight the limitations of widely used models, such as mode collapse. We also demonstrate the importance of rank metrics which complement traditional model fit measures, such as RMSE, for validating model effectiveness. Notably, our results show that while no single model architecture clearly outperforms others, simpler architectures are generally competitive and scale well with larger datasets. Overall, this benchmarking exercise sets new standards for model evaluation, supports robust model development, and advances the potential of these models to use high-throughput genetic and chemical screens for disease target discovery.","url_abs":"https://arxiv.org/abs/2408.10609v3","url_pdf":"https://arxiv.org/pdf/2408.10609v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"perturbench-benchmarking-machine-learning","repo_url":"https://github.com/altoslabs/perturbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.10609","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.10609"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/altoslabs/perturbench","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":2,"unverified":7},"by_repo_kind":{"official":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"a7c6b1b483541269","entry":"load_data_split","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/analysis/embed_cells.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/analysis/embed_cells.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a7c6b1b483541269"}},{"code_sha256_prefix":"8c33f01afd89956c","entry":"merge_cols","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/analysis/utils.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/analysis/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8c33f01afd89956c"}},{"code_sha256_prefix":"6773d7b436f91ed7","entry":"download_scgpt_model","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/analysis/embed_cells.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/analysis/embed_cells.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6773d7b436f91ed7"}},{"code_sha256_prefix":"ddadf26c0a2f0d74","entry":"embed_scgpt","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/analysis/embed_cells.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/analysis/embed_cells.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ddadf26c0a2f0d74"}},{"code_sha256_prefix":"333f12309c503fa0","entry":"instantiate_with_context","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/modelcore/utils.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/modelcore/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"333f12309c503fa0"}},{"code_sha256_prefix":"db043541e9083d80","entry":"load_eval","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/analysis/benchmarks/evaluation.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/analysis/benchmarks/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"db043541e9083d80"}},{"code_sha256_prefix":"04701ae765ae5215","entry":"multi_instantiate","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/modelcore/utils.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/modelcore/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"04701ae765ae5215"}},{"code_sha256_prefix":"e18c928222f26901","entry":"pert_cluster_filter","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/analysis/utils.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/analysis/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e18c928222f26901"}},{"code_sha256_prefix":"24193b5708258462","entry":"subsample_by_group","repo":"altoslabs/perturbench","repo_kind":"official","path":"src/perturbench/analysis/utils.py","file_url":"https://github.com/altoslabs/perturbench/blob/HEAD/src/perturbench/analysis/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"24193b5708258462"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}