{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-mabe22-benchmarks-for-representation","title":"MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior","arxiv_id":"2207.10553","date":"2022-07-21","proceeding":null,"authors":["Jennifer J. Sun","Markus Marks","Andrew Ulmer","Dipam Chakraborty","Brian Geuther","Edward Hayes","Heng Jia","Vivek Kumar","Sebastian Oleszko","Zachary Partridge","Milan Peelman","Alice Robie","Catherine E. Schretter","Keith Sheppard","Chao Sun","Param Uttarwar","Julian M. Wagner","Eric Werner","Joseph Parker","Pietro Perona","Yisong Yue","Kristin Branson","Ann Kennedy"],"abstract":"We introduce MABe22, a large-scale, multi-agent video and trajectory benchmark to assess the quality of learned behavior representations. This dataset is collected from a variety of biology experiments, and includes triplets of interacting mice (4.7 million frames video+pose tracking data, 10 million frames pose only), symbiotic beetle-ant interactions (10 million frames video data), and groups of interacting flies (4.4 million frames of pose tracking data). Accompanying these data, we introduce a panel of real-life downstream analysis tasks to assess the quality of learned representations by evaluating how well they preserve information about the experimental conditions (e.g. strain, time of day, optogenetic stimulation) and animal behavior. We test multiple state-of-the-art self-supervised video and trajectory representation learning methods to demonstrate the use of our benchmark, revealing that methods developed using human action datasets do not fully translate to animal datasets. We hope that our benchmark and dataset encourage a broader exploration of behavior representation learning methods across species and settings.","url_abs":"https://arxiv.org/abs/2207.10553v2","url_pdf":"https://arxiv.org/pdf/2207.10553v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-mabe22-benchmarks-for-representation","repo_url":"https://github.com/irlab-therapeutics/mabe_2022","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"pose-tracking","task_name":"Pose Tracking"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2207.10553","atlas_url":"https://app.syntology.ai/?focus=2207.10553","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.10553"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/IRLAB-Therapeutics/mabe_2022","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/JiaHeng-DLUT/MABe2022","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Param-Uttarwar/mabe","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/edhayes1/MABe","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/KumarLabJax/JABS-data-pipeline","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/irlab-therapeutics/mabe_2022","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"b6f830514e9874b0","entry":"create_bounding_box","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/video_dataset.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/video_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6f830514e9874b0"}},{"code_sha256_prefix":"f1b4d733b8e23f8c","entry":"encode_angle","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/handcrafted_geometries.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/handcrafted_geometries.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1b4d733b8e23f8c"}},{"code_sha256_prefix":"c6f9d81c41c5531f","entry":"find_corners","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/find_cage_features.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/find_cage_features.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6f9d81c41c5531f"}},{"code_sha256_prefix":"3acb200f06ba0194","entry":"find_edge","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/find_cage_features.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/find_cage_features.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3acb200f06ba0194"}},{"code_sha256_prefix":"527d57cb512bcc8e","entry":"find_intersection","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/find_cage_features.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/find_cage_features.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"527d57cb512bcc8e"}},{"code_sha256_prefix":"673c8d94f0ecf918","entry":"get_keypoints_map","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/keypoint_dataset.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/keypoint_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"673c8d94f0ecf918"}},{"code_sha256_prefix":"3c1d9542aa38d20e","entry":"polygon_area","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/handcrafted_geometries.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/handcrafted_geometries.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c1d9542aa38d20e"}},{"code_sha256_prefix":"9084513f06fedbf0","entry":"validate_submission","repo":"JiaHeng-DLUT/MABe2022","repo_kind":"found_in_text","path":"mabe/models/moco_model.py","file_url":"https://github.com/JiaHeng-DLUT/MABe2022/blob/HEAD/mabe/models/moco_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9084513f06fedbf0"}},{"code_sha256_prefix":"93ca5cca72ba494a","entry":"normalize_mouse","repo":"IRLAB-Therapeutics/mabe_2022","repo_kind":"official","path":"utils/handcrafted_geometries.py","file_url":"https://github.com/IRLAB-Therapeutics/mabe_2022/blob/HEAD/utils/handcrafted_geometries.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93ca5cca72ba494a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}