{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/navgpt-2-unleashing-navigational-reasoning","title":"NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models","arxiv_id":"2407.12366","date":"2024-07-17","proceeding":null,"authors":["Gengze Zhou","Yicong Hong","Zun Wang","Xin Eric Wang","Qi Wu"],"abstract":"Capitalizing on the remarkable advancements in Large Language Models (LLMs), there is a burgeoning initiative to harness LLMs for instruction following robotic navigation. Such a trend underscores the potential of LLMs to generalize navigational reasoning and diverse language understanding. However, a significant discrepancy in agent performance is observed when integrating LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous downstream specialist models. Furthermore, the inherent capacity of language to interpret and facilitate communication in agent interactions is often underutilized in these integrations. In this work, we strive to bridge the divide between VLN-specialized models and LLM-based navigation paradigms, while maintaining the interpretative prowess of LLMs in generating linguistic navigational reasoning. By aligning visual content in a frozen LLM, we encompass visual observation comprehension for LLMs and exploit a way to incorporate LLMs and navigation policy networks for effective action predictions and navigational reasoning. We demonstrate the data efficiency of the proposed methods and eliminate the gap between LM-based agents and state-of-the-art VLN specialists.","url_abs":"https://arxiv.org/abs/2407.12366v2","url_pdf":"https://arxiv.org/pdf/2407.12366v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"navgpt-2-unleashing-navigational-reasoning","repo_url":"https://github.com/gengzezhou/navgpt-2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.12366","atlas_url":"https://app.syntology.ai/?focus=2407.12366","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.12366"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gengzezhou/navgpt-2","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/GengzeZhou/NavGPT-2","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":7,"ran_violates":1,"ran_fixture":2,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":14,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/NavGPT_model.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/NavGPT_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"cbef633255b01324","entry":"cal_cls","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/r2r/eval_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/r2r/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbef633255b01324"}},{"code_sha256_prefix":"7ada68c978baa7c9","entry":"cal_dtw","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/r2r/eval_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/r2r/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ada68c978baa7c9"}},{"code_sha256_prefix":"c362aaed2935ceff","entry":"calc_position_distance","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/graph_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/graph_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c362aaed2935ceff"}},{"code_sha256_prefix":"168e35f8ed1ad0ce","entry":"calculate_vp_rel_pos_fts","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/graph_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/graph_utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"168e35f8ed1ad0ce"}},{"code_sha256_prefix":"eb5d2aab5fa48a08","entry":"construct_instrs","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/r2r/data_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/r2r/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb5d2aab5fa48a08"}},{"code_sha256_prefix":"91e1742acf23aa57","entry":"extend_neg_masks","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/ops.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/ops.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91e1742acf23aa57"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/NavGPT_model.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/NavGPT_model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"d9c76c6dc9976905","entry":"gen_seq_masks","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/ops.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/ops.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9c76c6dc9976905"}},{"code_sha256_prefix":"dac1993f5ce8e12e","entry":"get_angle_fts","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/graph_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/graph_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dac1993f5ce8e12e"}},{"code_sha256_prefix":"80881fb0e12a6a8e","entry":"load_instr_datasets","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/r2r/data_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/r2r/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80881fb0e12a6a8e"}},{"code_sha256_prefix":"ab1a50dceb5369de","entry":"load_obj2vps","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/r2r/data_utils.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/r2r/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab1a50dceb5369de"}},{"code_sha256_prefix":"df2919fd1dad1d43","entry":"build_transformer","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/transformer.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df2919fd1dad1d43"}},{"code_sha256_prefix":"c4d81965c22f62f9","entry":"create_transformer_encoder","repo":"GengzeZhou/NavGPT-2","repo_kind":"official","path":"map_nav_src/models/ops.py","file_url":"https://github.com/GengzeZhou/NavGPT-2/blob/HEAD/map_nav_src/models/ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4d81965c22f62f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}