{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-embodied-generalist-agent-in-3d-world","title":"An Embodied Generalist Agent in 3D World","arxiv_id":"2311.12871","date":"2023-11-18","proceeding":null,"authors":["Jiangyong Huang","Silong Yong","Xiaojian Ma","Xiongkun Linghu","Puhao Li","Yan Wang","Qing Li","Song-Chun Zhu","Baoxiong Jia","Siyuan Huang"],"abstract":"Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising diverse object-level and scene-level tasks, which require considerable understanding of and interaction with the 3D world. Moreover, we meticulously design an LLM-assisted pipeline to produce high-quality 3D VL data. Through extensive experiments, we demonstrate LEO's remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, navigation and manipulation. Our ablative studies and scaling analyses further provide valuable insights for developing future embodied generalist agents. Code and data are available on project page.","url_abs":"https://arxiv.org/abs/2311.12871v3","url_pdf":"https://arxiv.org/pdf/2311.12871v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-embodied-generalist-agent-in-3d-world","repo_url":"https://github.com/embodied-generalist/embodied-generalist","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-question-answering-3d-qa","task_name":"3D Question Answering (3D-QA)"},{"task_slug":"3d-dense-captioning","task_name":"3D dense captioning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"},{"task_slug":"scene-aware-dialogue","task_name":"Scene-Aware Dialogue"},{"task_slug":"vision-language-navigation","task_name":"Vision-Language Navigation"},{"task_slug":"vision-language-action","task_name":"Vision-Language-Action"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-question-answering-3d-qa-on-sqa3d","task":"3D Question Answering (3D-QA)","dataset":"SQA3D","model":"LEO","rank_in_archive_order":6,"of":13,"metrics":{"Exact Match":"50.0"},"uses_additional_data":false},{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"LEO","rank_in_archive_order":6,"of":18,"metrics":{"BLEU-4":"13.2","CIDEr":"101.4","Exact Match":"24.5","METEOR":"20.0","ROUGE":"49.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.12871","atlas_url":"https://app.syntology.ai/?focus=2311.12871","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.12871"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/embodied-generalist/embodied-generalist","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":1,"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":13,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9a65a30d006fc96e","entry":"apply_rotary_pos_emb","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"model/transformers.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/model/transformers.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a65a30d006fc96e"}},{"code_sha256_prefix":"592aacb855a01d0f","entry":"cfg2dict","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"common/type_utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/common/type_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"592aacb855a01d0f"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"model/transformers.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/model/transformers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"9d2100678022a09b","entry":"disabled_train","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/model/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d2100678022a09b"}},{"code_sha256_prefix":"edefc587492eb250","entry":"generate_fourier_features","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"model/vision3d.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/model/vision3d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edefc587492eb250"}},{"code_sha256_prefix":"165417eb35c55e89","entry":"get_activation_fn","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/model/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"165417eb35c55e89"}},{"code_sha256_prefix":"2fb14462bf1a1166","entry":"is_port_available","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"common/launch_utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/common/launch_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fb14462bf1a1166"}},{"code_sha256_prefix":"1eee3af921bc5d88","entry":"load_imgs","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"common/io_utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/common/io_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1eee3af921bc5d88"}},{"code_sha256_prefix":"bace1acf5a748097","entry":"load_json","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"common/io_utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/common/io_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bace1acf5a748097"}},{"code_sha256_prefix":"7de9d66696a307eb","entry":"load_pickle","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"common/io_utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/common/io_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7de9d66696a307eb"}},{"code_sha256_prefix":"76f3fbdda12ce4eb","entry":"maybe_autocast","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/model/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76f3fbdda12ce4eb"}},{"code_sha256_prefix":"44925cf3831bfe79","entry":"rgetattr","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"common/misc.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/common/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44925cf3831bfe79"}},{"code_sha256_prefix":"f9de4cdd8a624801","entry":"rsetattr","repo":"embodied-generalist/embodied-generalist","repo_kind":"official","path":"common/misc.py","file_url":"https://github.com/embodied-generalist/embodied-generalist/blob/HEAD/common/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9de4cdd8a624801"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}