{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-colosseum-a-benchmark-for-evaluating","title":"THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation","arxiv_id":"2402.08191","date":"2024-02-13","proceeding":null,"authors":["Wilbert Pumacay","Ishika Singh","Jiafei Duan","Ranjay Krishna","Jesse Thomason","Dieter Fox"],"abstract":"To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a majority of studies evaluate robot performance in environments closely resembling or even identical to the training setup. We present THE COLOSSEUM, a novel simulation benchmark, with 20 diverse manipulation tasks, that enables systematical evaluation of models across 14 axes of environmental perturbations. These perturbations include changes in color, texture, and size of objects, table-tops, and backgrounds; we also vary lighting, distractors, physical properties perturbations and camera pose. Using THE COLOSSEUM, we compare 5 state-of-the-art manipulation models to reveal that their success rate degrades between 30-50% across these perturbation factors. When multiple perturbations are applied in unison, the success rate degrades $\\geq$75%. We identify that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most. To verify the ecological validity of our results, we show that our results in simulation are correlated ($\\bar{R}^2 = 0.614$) to similar perturbations in real-world experiments. We open source code for others to use THE COLOSSEUM, and also release code to 3D print the objects used to replicate the real-world perturbations. Ultimately, we hope that THE COLOSSEUM will serve as a benchmark to identify modeling decisions that systematically improve generalization for manipulation. See https://robot-colosseum.github.io/ for more details.","url_abs":"https://arxiv.org/abs/2402.08191v2","url_pdf":"https://arxiv.org/pdf/2402.08191v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-colosseum-a-benchmark-for-evaluating","repo_url":"https://github.com/robot-colosseum/robot-colosseum","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"robot-manipulation-generalization","task_name":"Robot Manipulation Generalization"}],"methods":[],"datasets_introduced":[{"slug":"the-colosseum","name":"The COLOSSEUM","full_name":"The COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.08191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.08191"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/robot-colosseum/robot-colosseum","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"922ff1cd170aa462","entry":"get_variation_name","repo":"robot-colosseum/robot-colosseum","repo_kind":"official","path":"colosseum/rlbench/utils.py","file_url":"https://github.com/robot-colosseum/robot-colosseum/blob/HEAD/colosseum/rlbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"922ff1cd170aa462"}},{"code_sha256_prefix":"fdf937b7b79c757a","entry":"sampleFriction","repo":"robot-colosseum/robot-colosseum","repo_kind":"official","path":"colosseum/variations/object_friction.py","file_url":"https://github.com/robot-colosseum/robot-colosseum/blob/HEAD/colosseum/variations/object_friction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdf937b7b79c757a"}},{"code_sha256_prefix":"3f83ebf7a759c731","entry":"sampleMass","repo":"robot-colosseum/robot-colosseum","repo_kind":"official","path":"colosseum/variations/object_mass.py","file_url":"https://github.com/robot-colosseum/robot-colosseum/blob/HEAD/colosseum/variations/object_mass.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f83ebf7a759c731"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}