{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arnold-a-benchmark-for-language-grounded-task","title":"ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes","arxiv_id":"2304.04321","date":"2023-04-09","proceeding":"ICCV 2023 1","authors":["Ran Gong","Jiangyong Huang","Yizhou Zhao","Haoran Geng","Xiaofeng Gao","Qingyang Wu","Wensi Ai","Ziheng Zhou","Demetri Terzopoulos","Song-Chun Zhu","Baoxiong Jia","Siyuan Huang"],"abstract":"Understanding the continuous states of objects is essential for task learning and planning in the real world. However, most existing task learning benchmarks assume discrete (e.g., binary) object goal states, which poses challenges for the learning of complex tasks and transferring learned policy from simulated environments to the real world. Furthermore, state discretization limits a robot's ability to follow human instructions based on the grounding of actions and states. To tackle these challenges, we present ARNOLD, a benchmark that evaluates language-grounded task learning with continuous states in realistic 3D scenes. ARNOLD is comprised of 8 language-conditioned tasks that involve understanding object states and learning policies for continuous goals. To promote language-instructed learning, we provide expert demonstrations with template-generated language descriptions. We assess task performance by utilizing the latest language-conditioned policy learning models. Our results indicate that current models for language-conditioned manipulations continue to experience significant challenges in novel goal-state generalizations, scene generalizations, and object generalizations. These findings highlight the need to develop new algorithms that address this gap and underscore the potential for further research in this area. Project website: https://arnold-benchmark.github.io.","url_abs":"https://arxiv.org/abs/2304.04321v2","url_pdf":"https://arxiv.org/pdf/2304.04321v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"arnold-a-benchmark-for-language-grounded-task","repo_url":"https://github.com/arnold-benchmark/arnold","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"object","task_name":"Object"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.04321","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.04321"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/arnold-benchmark/arnold","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":1,"ran_violates":2,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c99fb7b9654ea364","entry":"act_layer","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"bc_z/blocks.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/bc_z/blocks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c99fb7b9654ea364"}},{"code_sha256_prefix":"8458f9ba0361ff43","entry":"cache_fn","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"peract/network.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/peract/network.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8458f9ba0361ff43"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"peract/network.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/peract/network.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"peract/network.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/peract/network.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"9ed9933be23b28a5","entry":"norm_layer1d","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"bc_z/blocks.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/bc_z/blocks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ed9933be23b28a5"}},{"code_sha256_prefix":"d0264b5831943fb9","entry":"norm_layer2d","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"bc_z/blocks.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/bc_z/blocks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0264b5831943fb9"}},{"code_sha256_prefix":"0800b4fef63dbf21","entry":"load_ckpt","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"ckpt_selection.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/ckpt_selection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0800b4fef63dbf21"}},{"code_sha256_prefix":"e3f04ff9250f8f3f","entry":"load_data","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"ckpt_selection.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/ckpt_selection.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3f04ff9250f8f3f"}},{"code_sha256_prefix":"bab389ab808463dc","entry":"parse_state","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bab389ab808463dc"}},{"code_sha256_prefix":"4e87462cc7d896d2","entry":"sec_to_str","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"train_cliport6d.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/train_cliport6d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e87462cc7d896d2"}},{"code_sha256_prefix":"e241a1dff2e417df","entry":"val","repo":"arnold-benchmark/arnold","repo_kind":"official","path":"train_cliport6d.py","file_url":"https://github.com/arnold-benchmark/arnold/blob/HEAD/train_cliport6d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e241a1dff2e417df"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}