{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/embodied-agent-interface-benchmarking-llms","title":"Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making","arxiv_id":"2410.07166","date":"2024-10-09","proceeding":null,"authors":["Manling Li","Shiyu Zhao","Qineng Wang","Kangrui Wang","Yu Zhou","Sanjana Srivastava","Cem Gokmen","Tony Lee","Li Erran Li","Ruohan Zhang","Weiyu Liu","Percy Liang","Li Fei-Fei","Jiayuan Mao","Jiajun Wu"],"abstract":"We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.","url_abs":"https://arxiv.org/abs/2410.07166v3","url_pdf":"https://arxiv.org/pdf/2410.07166v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"embodied-agent-interface-benchmarking-llms","repo_url":"https://github.com/embodied-agent-eval/embodied-agent-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"embodied-agent-interface-benchmarking-llms","repo_url":"https://github.com/embodied-agent-interface/embodied-agent-interface","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"embodied-agent-interface-benchmarking-llms","repo_url":"https://github.com/embodied-agent-eval/embodied-agent-eval.github.io","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.07166","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.07166"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/embodied-agent-eval/embodied-agent-eval.github.io","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/embodied-agent-eval/embodied-agent-eval","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/embodied-agent-interface/embodied-agent-interface","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"280f43cf0bb2fce8","entry":"build_id_to_name_dict","repo":"embodied-agent-eval/embodied-agent-eval","repo_kind":"official","path":"src/behavior_eval/tl_formula/simple_tl.py","file_url":"https://github.com/embodied-agent-eval/embodied-agent-eval/blob/HEAD/src/behavior_eval/tl_formula/simple_tl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"280f43cf0bb2fce8"}},{"code_sha256_prefix":"30950b04bb176778","entry":"build_name_to_id_list_dict","repo":"embodied-agent-eval/embodied-agent-eval","repo_kind":"official","path":"src/behavior_eval/tl_formula/simple_tl.py","file_url":"https://github.com/embodied-agent-eval/embodied-agent-eval/blob/HEAD/src/behavior_eval/tl_formula/simple_tl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30950b04bb176778"}},{"code_sha256_prefix":"a3ca20d9bcdffb24","entry":"get_first_object_id","repo":"embodied-agent-eval/embodied-agent-eval","repo_kind":"official","path":"src/behavior_eval/tl_formula/simple_tl.py","file_url":"https://github.com/embodied-agent-eval/embodied-agent-eval/blob/HEAD/src/behavior_eval/tl_formula/simple_tl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3ca20d9bcdffb24"}},{"code_sha256_prefix":"7147db4dbc131be3","entry":"replace_wildcard_name","repo":"embodied-agent-eval/embodied-agent-eval","repo_kind":"official","path":"src/behavior_eval/tl_formula/bddl_to_tl.py","file_url":"https://github.com/embodied-agent-eval/embodied-agent-eval/blob/HEAD/src/behavior_eval/tl_formula/bddl_to_tl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7147db4dbc131be3"}},{"code_sha256_prefix":"3fd0ee4a843d495d","entry":"setup_logging","repo":"embodied-agent-eval/embodied-agent-eval","repo_kind":"official","path":"src/virtualhome_eval/log_config.py","file_url":"https://github.com/embodied-agent-eval/embodied-agent-eval/blob/HEAD/src/virtualhome_eval/log_config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fd0ee4a843d495d"}},{"code_sha256_prefix":"93c1d9a8e4d41929","entry":"translate_addressable_obj_into_tl_obj","repo":"embodied-agent-eval/embodied-agent-eval","repo_kind":"official","path":"src/behavior_eval/tl_formula/bddl_to_tl.py","file_url":"https://github.com/embodied-agent-eval/embodied-agent-eval/blob/HEAD/src/behavior_eval/tl_formula/bddl_to_tl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93c1d9a8e4d41929"}},{"code_sha256_prefix":"e2fa5cdda3908872","entry":"translate_tl_obj_into_addressable_obj","repo":"embodied-agent-eval/embodied-agent-eval","repo_kind":"official","path":"src/behavior_eval/tl_formula/bddl_to_tl.py","file_url":"https://github.com/embodied-agent-eval/embodied-agent-eval/blob/HEAD/src/behavior_eval/tl_formula/bddl_to_tl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2fa5cdda3908872"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}