{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/navcot-boosting-llm-based-vision-and-language","title":"NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning","arxiv_id":"2403.07376","date":"2024-03-12","proceeding":null,"authors":["Bingqian Lin","Yunshuang Nie","Ziming Wei","Jiaqi Chen","Shikui Ma","Jianhua Han","Hang Xu","Xiaojun Chang","Xiaodan Liang"],"abstract":"Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions. Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability. However, their predominant use in an offline manner usually suffers from substantial domain gap between the VLN task and the LLM training corpus. This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision, leading to a significant mitigation of the domain gap in a cost-effective manner. Specifically, at each timestep, the LLM is prompted to forecast the navigational chain-of-thought by: 1) acting as a world model to imagine the next observation according to the instruction, 2) selecting the candidate observation that best aligns with the imagination, and 3) determining the action based on the reasoning from the prior steps. Through constructing formalized labels for training, the LLM can learn to generate desired and reasonable chain-of-thought outputs for improving the action decision. Experimental results across various training settings and popular VLN benchmarks (e.g., Room-to-Room (R2R), Room-across-Room (RxR), Room-for-Room (R4R)) show the significant superiority of NavCoT over the direct action prediction variants. Through simple parameter-efficient finetuning, our NavCoT outperforms a recent GPT4-based approach with ~7% relative improvement on the R2R dataset. We believe that NavCoT will help unlock more task-adaptive and scalable LLM-based embodied agents, which are helpful for developing real-world robotics applications. Code is available at https://github.com/expectorlin/NavCoT.","url_abs":"https://arxiv.org/abs/2403.07376v2","url_pdf":"https://arxiv.org/pdf/2403.07376v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"navcot-boosting-llm-based-vision-and-language","repo_url":"https://github.com/expectorlin/navcot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.07376","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.07376"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/expectorlin/navcot","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":5,"ran_honours":1,"unverified":5},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/models/vilmodel_cmt.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/models/vilmodel_cmt.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"cbef633255b01324","entry":"cal_cls","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/r2r/eval_utils.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/r2r/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbef633255b01324"}},{"code_sha256_prefix":"7ada68c978baa7c9","entry":"cal_dtw","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/r2r/eval_utils.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/r2r/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ada68c978baa7c9"}},{"code_sha256_prefix":"8423db639d15cbb7","entry":"construct_instrs","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/r2r/data_utils.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/r2r/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8423db639d15cbb7"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/models/vilmodel_cmt.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/models/vilmodel_cmt.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"34472603c7e3ebfc","entry":"is_default_gpu","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/utils/distributed.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/utils/distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34472603c7e3ebfc"}},{"code_sha256_prefix":"635ca59e339f7321","entry":"load_nav_graphs","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/r2r/data_utils.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/r2r/data_utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"635ca59e339f7321"}},{"code_sha256_prefix":"ca9190060fcbf372","entry":"postprocess_args","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/r2r/parser.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/r2r/parser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca9190060fcbf372"}},{"code_sha256_prefix":"f37a7cdf5beda0f8","entry":"get_tokenizer","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/models/vlnbert_init.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/models/vlnbert_init.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f37a7cdf5beda0f8"}},{"code_sha256_prefix":"67fb9db398d1fe73","entry":"get_vlnbert_models","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/models/vlnbert_init.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/models/vlnbert_init.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67fb9db398d1fe73"}},{"code_sha256_prefix":"8638e244b7ad3376","entry":"init_distributed","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/utils/distributed.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/utils/distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8638e244b7ad3376"}},{"code_sha256_prefix":"e04e867548dfe3e7","entry":"load_init_param","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/utils/distributed.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/utils/distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e04e867548dfe3e7"}},{"code_sha256_prefix":"20f09ba1108aefe4","entry":"load_instr_datasets","repo":"expectorlin/navcot","repo_kind":"official","path":"finetune_src/r2r/data_utils.py","file_url":"https://github.com/expectorlin/navcot/blob/HEAD/finetune_src/r2r/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20f09ba1108aefe4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}