{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/embodied-multi-modal-agent-trained-by-an-llm","title":"Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld","arxiv_id":"2311.16714","date":"2023-11-28","proceeding":"CVPR 2024 1","authors":["Yijun Yang","Tianyi Zhou","Kanxue Li","Dapeng Tao","Lusong Li","Li Shen","Xiaodong He","Jing Jiang","Yuhui Shi"],"abstract":"While large language models (LLMs) excel in a simulated world of texts, they struggle to interact with the more realistic world without perceptions of other modalities such as visual or audio signals. Although vision-language models (VLMs) integrate LLM modules (1) aligned with static image features, and (2) may possess prior knowledge of world dynamics (as demonstrated in the text world), they have not been trained in an embodied visual world and thus cannot align with its dynamics. On the other hand, training an embodied agent in a noisy visual world without expert guidance is often challenging and inefficient. In this paper, we train a VLM agent living in a visual world using an LLM agent excelling in a parallel text world. Specifically, we distill LLM's reflection outcomes (improved actions by analyzing mistakes) in a text world's tasks to finetune the VLM on the same tasks of the visual world, resulting in an Embodied Multi-Modal Agent (EMMA) quickly adapting to the visual world dynamics. Such cross-modality imitation learning between the two parallel worlds is achieved by a novel DAgger-DPO algorithm, enabling EMMA to generalize to a broad scope of new tasks without any further guidance from the LLM expert. Extensive evaluations on the ALFWorld benchmark's diverse tasks highlight EMMA's superior performance to SOTA VLM-based agents, e.g., 20%-70% improvement in the success rate.","url_abs":"https://arxiv.org/abs/2311.16714v2","url_pdf":"https://arxiv.org/pdf/2311.16714v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"embodied-multi-modal-agent-trained-by-an-llm","repo_url":"https://github.com/stevenyangyj/emma-alfworld","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.16714","atlas_url":"https://app.syntology.ai/?focus=2311.16714","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.16714"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stevenyangyj/emma-alfworld","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"b3f1ce8a96e4feb4","entry":"check_put","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/post_processing.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/post_processing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3f1ce8a96e4feb4"}},{"code_sha256_prefix":"112c839f24908701","entry":"dpo_loss","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/lavis/models/blip2_models/blip2_emma.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/lavis/models/blip2_models/blip2_emma.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"112c839f24908701"}},{"code_sha256_prefix":"6a7cb4ca218f011f","entry":"remove_the","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/post_processing.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/post_processing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a7cb4ca218f011f"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/lavis/models/base_model.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/lavis/models/base_model.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"ebee9e987c95636b","entry":"get_completion","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/emma_utils.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/emma_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebee9e987c95636b"}},{"code_sha256_prefix":"a7ece9f49be6eb57","entry":"num_tokens_from_messages","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/emma_utils.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/emma_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a7ece9f49be6eb57"}},{"code_sha256_prefix":"918526f0d411a7aa","entry":"num_tokens_from_string","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/emma_utils.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/emma_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"918526f0d411a7aa"}},{"code_sha256_prefix":"dfb17eb3e6eaed35","entry":"process_ob","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/post_processing.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/post_processing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfb17eb3e6eaed35"}},{"code_sha256_prefix":"bad8273de604ee8a","entry":"update_memory","repo":"stevenyangyj/emma-alfworld","repo_kind":"official","path":"LAVIS/generate_reflections.py","file_url":"https://github.com/stevenyangyj/emma-alfworld/blob/HEAD/LAVIS/generate_reflections.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bad8273de604ee8a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}