{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lacma-language-aligning-contrastive-learning","title":"LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following","arxiv_id":"2310.12344","date":"2023-10-18","proceeding":null,"authors":["Cheng-Fu Yang","Yen-Chun Chen","Jianwei Yang","Xiyang Dai","Lu Yuan","Yu-Chiang Frank Wang","Kai-Wei Chang"],"abstract":"End-to-end Transformers have demonstrated an impressive success rate for Embodied Instruction Following when the environment has been seen in training. However, they tend to struggle when deployed in an unseen environment. This lack of generalizability is due to the agent's insensitivity to subtle changes in natural language instructions. To mitigate this issue, we propose explicitly aligning the agent's hidden states with the instructions via contrastive learning. Nevertheless, the semantic gap between high-level language instructions and the agent's low-level action space remains an obstacle. Therefore, we further introduce a novel concept of meta-actions to bridge the gap. Meta-actions are ubiquitous action patterns that can be parsed from the original action sequence. These patterns represent higher-level semantics that are intuitively aligned closer to the instructions. When meta-actions are applied as additional training signals, the agent generalizes better to unseen environments. Compared to a strong multi-modal Transformer baseline, we achieve a significant 4.5% absolute gain in success rate in unseen environments of ALFRED Embodied Instruction Following. Additional analysis shows that the contrastive objective and meta-actions are complementary in achieving the best results, and the resulting agent better aligns its states with corresponding instructions, making it more suitable for real-world embodied agents. The code is available at: https://github.com/joeyy5588/LACMA.","url_abs":"https://arxiv.org/abs/2310.12344v1","url_pdf":"https://arxiv.org/pdf/2310.12344v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lacma-language-aligning-contrastive-learning","repo_url":"https://github.com/joeyy5588/lacma","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.12344","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12344"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/joeyy5588/LACMA","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4f627def4b9d8ff7","entry":"create_optimizer_and_schedulers","repo":"joeyy5588/LACMA","repo_kind":"official","path":"alfred/utils/model_util.py","file_url":"https://github.com/joeyy5588/LACMA/blob/HEAD/alfred/utils/model_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f627def4b9d8ff7"}},{"code_sha256_prefix":"12ba53fedf724ead","entry":"get_action","repo":"joeyy5588/LACMA","repo_kind":"official","path":"alfred/env/reward.py","file_url":"https://github.com/joeyy5588/LACMA/blob/HEAD/alfred/env/reward.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12ba53fedf724ead"}},{"code_sha256_prefix":"d11401259503326f","entry":"get_task","repo":"joeyy5588/LACMA","repo_kind":"official","path":"alfred/env/tasks.py","file_url":"https://github.com/joeyy5588/LACMA/blob/HEAD/alfred/env/tasks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d11401259503326f"}},{"code_sha256_prefix":"026a1fec14fca72d","entry":"load_model","repo":"joeyy5588/LACMA","repo_kind":"official","path":"alfred/utils/model_util.py","file_url":"https://github.com/joeyy5588/LACMA/blob/HEAD/alfred/utils/model_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"026a1fec14fca72d"}},{"code_sha256_prefix":"d34e3ef19764c321","entry":"load_model_args","repo":"joeyy5588/LACMA","repo_kind":"official","path":"alfred/utils/model_util.py","file_url":"https://github.com/joeyy5588/LACMA/blob/HEAD/alfred/utils/model_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d34e3ef19764c321"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}