{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sub-instruction-aware-vision-and-language","title":"Sub-Instruction Aware Vision-and-Language Navigation","arxiv_id":"2004.02707","date":"2020-04-06","proceeding":"EMNLP 2020 11","authors":["Yicong Hong","Cristian Rodriguez-Opazo","Qi Wu","Stephen Gould"],"abstract":"Vision-and-language navigation requires an agent to navigate through a real 3D environment following natural language instructions. Despite significant advances, few previous works are able to fully utilize the strong correspondence between the visual and textual sequences. Meanwhile, due to the lack of intermediate supervision, the agent's performance at following each part of the instruction cannot be assessed during navigation. In this work, we focus on the granularity of the visual and language sequences as well as the traceability of agents through the completion of an instruction. We provide agents with fine-grained annotations during training and find that they are able to follow the instruction better and have a higher chance of reaching the target at test time. We enrich the benchmark dataset Room-to-Room (R2R) with sub-instructions and their corresponding paths. To make use of this data, we propose effective sub-instruction attention and shifting modules that select and attend to a single sub-instruction at each time-step. We implement our sub-instruction modules in four state-of-the-art agents, compare with their baseline models, and show that our proposed method improves the performance of all four agents. We release the Fine-Grained R2R dataset (FGR2R) and the code at https://github.com/YicongHong/Fine-Grained-R2R.","url_abs":"https://arxiv.org/abs/2004.02707v2","url_pdf":"https://arxiv.org/pdf/2004.02707v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sub-instruction-aware-vision-and-language","repo_url":"https://github.com/YicongHong/Fine-Grained-R2R","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"}],"methods":[],"datasets_introduced":[{"slug":"fine-grained-r2r","name":"Fine-Grained R2R","full_name":null}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.02707","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.02707"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YicongHong/Fine-Grained-R2R","reach":null}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fd4e4d97b73c90f2","entry":"check_lemma","repo":"YicongHong/Fine-Grained-R2R","repo_kind":"official","path":"source/chunking_function.py","file_url":"https://github.com/YicongHong/Fine-Grained-R2R/blob/HEAD/source/chunking_function.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd4e4d97b73c90f2"}},{"code_sha256_prefix":"8f2ee6387d2fc4b3","entry":"create_chunk","repo":"YicongHong/Fine-Grained-R2R","repo_kind":"official","path":"source/chunking_function.py","file_url":"https://github.com/YicongHong/Fine-Grained-R2R/blob/HEAD/source/chunking_function.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f2ee6387d2fc4b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}