{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cryobench-diverse-and-challenging-datasets","title":"CryoBench: Diverse and challenging datasets for the heterogeneity problem in cryo-EM","arxiv_id":"2408.05526","date":"2024-08-10","proceeding":null,"authors":["Minkyu Jeon","Rishwanth Raghu","Miro Astore","Geoffrey Woollard","Ryan Feathers","Alkin Kaz","Sonya M. Hanson","Pilar Cossio","Ellen D. Zhong"],"abstract":"Cryo-electron microscopy (cryo-EM) is a powerful technique for determining high-resolution 3D biomolecular structures from imaging data. Its unique ability to capture structural variability has spurred the development of heterogeneous reconstruction algorithms that can infer distributions of 3D structures from noisy, unlabeled imaging data. Despite the growing number of advanced methods, progress in the field is hindered by the lack of standardized benchmarks with ground truth information and reliable validation metrics. Here, we introduce CryoBench, a suite of datasets, metrics, and benchmarks for heterogeneous reconstruction in cryo-EM. CryoBench includes five datasets representing different sources of heterogeneity and degrees of difficulty. These include conformational heterogeneity generated from designed motions of antibody complexes or sampled from a molecular dynamics simulation, as well as compositional heterogeneity from mixtures of ribosome assembly states or 100 common complexes present in cells. We then analyze state-of-the-art heterogeneous reconstruction tools, including neural and non-neural methods, assess their sensitivity to noise, and propose new metrics for quantitative evaluation. We hope that CryoBench will be a foundational resource for accelerating algorithmic development and evaluation in the cryo-EM and machine learning communities. Project page: https://cryobench.cs.princeton.edu.","url_abs":"https://arxiv.org/abs/2408.05526v2","url_pdf":"https://arxiv.org/pdf/2408.05526v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cryobench-diverse-and-challenging-datasets","repo_url":"https://github.com/ml-struct-bio/cryobench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"3d-reconstruction","task_name":"3D Reconstruction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.05526","atlas_url":"https://app.syntology.ai/?focus=2408.05526","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.05526"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-struct-bio/cryobench","reach":null}],"summary":{"ran_fixture":2,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"995babec552fa384","entry":"add_ctf","repo":"ml-struct-bio/cryobench","repo_kind":"official","path":"cryosim/add_ctf.py","file_url":"https://github.com/ml-struct-bio/cryobench/blob/HEAD/cryosim/add_ctf.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"995babec552fa384"}},{"code_sha256_prefix":"7dac7a0f0d615414","entry":"add_noise","repo":"ml-struct-bio/cryobench","repo_kind":"official","path":"cryosim/add_ctf.py","file_url":"https://github.com/ml-struct-bio/cryobench/blob/HEAD/cryosim/add_ctf.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7dac7a0f0d615414"}},{"code_sha256_prefix":"d53a2550f4900bc5","entry":"natural_sort_key","repo":"ml-struct-bio/cryobench","repo_kind":"official","path":"metrics/visualization/visualize_umap_IgG-RL.py","file_url":"https://github.com/ml-struct-bio/cryobench/blob/HEAD/metrics/visualization/visualize_umap_IgG-RL.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d53a2550f4900bc5"}},{"code_sha256_prefix":"5276062bd81abca3","entry":"parse_class_abinit_assignments","repo":"ml-struct-bio/cryobench","repo_kind":"official","path":"metrics/visualization/visualize_umap_IgG-RL.py","file_url":"https://github.com/ml-struct-bio/cryobench/blob/HEAD/metrics/visualization/visualize_umap_IgG-RL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5276062bd81abca3"}},{"code_sha256_prefix":"1d1bce7a9c6017ea","entry":"parse_class_assignments","repo":"ml-struct-bio/cryobench","repo_kind":"official","path":"metrics/visualization/visualize_umap_IgG-RL.py","file_url":"https://github.com/ml-struct-bio/cryobench/blob/HEAD/metrics/visualization/visualize_umap_IgG-RL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d1bce7a9c6017ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}