{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/general-scene-adaptation-for-vision-and","title":"General Scene Adaptation for Vision-and-Language Navigation","arxiv_id":"2501.17403","date":"2025-01-29","proceeding":null,"authors":["Haodong Hong","Yanyuan Qiao","Sen Wang","Jiajun Liu","Qi Wu"],"abstract":"Vision-and-Language Navigation (VLN) tasks mainly evaluate agents based on one-time execution of individual instructions across multiple environments, aiming to develop agents capable of functioning in any environment in a zero-shot manner. However, real-world navigation robots often operate in persistent environments with relatively consistent physical layouts, visual observations, and language styles from instructors. Such a gap in the task setting presents an opportunity to improve VLN agents by incorporating continuous adaptation to specific environments. To better reflect these real-world conditions, we introduce GSA-VLN, a novel task requiring agents to execute navigation instructions within a specific scene and simultaneously adapt to it for improved performance over time. To evaluate the proposed task, one has to address two challenges in existing VLN datasets: the lack of OOD data, and the limited number and style diversity of instructions for each scene. Therefore, we propose a new dataset, GSA-R2R, which significantly expands the diversity and quantity of environments and instructions for the R2R dataset to evaluate agent adaptability in both ID and OOD contexts. Furthermore, we design a three-stage instruction orchestration pipeline that leverages LLMs to refine speaker-generated instructions and apply role-playing techniques to rephrase instructions into different speaking styles. This is motivated by the observation that each individual user often has consistent signatures or preferences in their instructions. We conducted extensive experiments on GSA-R2R to thoroughly evaluate our dataset and benchmark various methods. Based on our findings, we propose a novel method, GR-DUET, which incorporates memory-based navigation graphs with an environment-specific training strategy, achieving state-of-the-art results on all GSA-R2R splits.","url_abs":"https://arxiv.org/abs/2501.17403v1","url_pdf":"https://arxiv.org/pdf/2501.17403v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"general-scene-adaptation-for-vision-and","repo_url":"https://github.com/honghd16/gsa-vln","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.17403","atlas_url":"https://app.syntology.ai/?focus=2501.17403","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.17403"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/honghd16/GSA-VLN","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/honghd16/gsa-vln","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"official":{"samples":14,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0feddf62036c64ee","entry":"TransformerEncoder","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0feddf62036c64ee"}},{"code_sha256_prefix":"e421ddf75512f4c4","entry":"create_transformer_encoder","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e421ddf75512f4c4"}},{"code_sha256_prefix":"c35512e4d250fe83","entry":"BertAttention","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c35512e4d250fe83"}},{"code_sha256_prefix":"66bd358094beb851","entry":"BertEmbeddings","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66bd358094beb851"}},{"code_sha256_prefix":"721f4cb09203dd0a","entry":"BertLayer","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"721f4cb09203dd0a"}},{"code_sha256_prefix":"50a3d34aac2573cd","entry":"BertSelfAttention","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50a3d34aac2573cd"}},{"code_sha256_prefix":"c26f9faf52e44089","entry":"ClsPrediction","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c26f9faf52e44089"}},{"code_sha256_prefix":"440e8824b1f7a4cc","entry":"CrossmodalEncoder","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"440e8824b1f7a4cc"}},{"code_sha256_prefix":"62f8ba1805bb2b0d","entry":"GlobalMapEncoder","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62f8ba1805bb2b0d"}},{"code_sha256_prefix":"367e98fa690b6937","entry":"GlocalTextPathNavCMT","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"367e98fa690b6937"}},{"code_sha256_prefix":"2445c8cf02121bbf","entry":"GraphLXRTXLayer","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2445c8cf02121bbf"}},{"code_sha256_prefix":"0d0612599d4b8d71","entry":"ImageEmbeddings","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d0612599d4b8d71"}},{"code_sha256_prefix":"ec6f57e27634d519","entry":"LanguageEncoder","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec6f57e27634d519"}},{"code_sha256_prefix":"5ddba8a53b37ee6f","entry":"LocalVPEncoder","repo":"honghd16/gsa-vln","repo_kind":"official","path":"GR-DUET/map_nav_src/models/vilmodel.py","file_url":"https://github.com/honghd16/gsa-vln/blob/HEAD/GR-DUET/map_nav_src/models/vilmodel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ddba8a53b37ee6f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}