{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-vision-and-language-navigation-from","title":"Learning Vision-and-Language Navigation from YouTube Videos","arxiv_id":"2307.11984","date":"2023-07-22","proceeding":"ICCV 2023 1","authors":["Kunyang Lin","Peihao Chen","Diwei Huang","Thomas H. Li","Mingkui Tan","Chuang Gan"],"abstract":"Vision-and-language navigation (VLN) requires an embodied agent to navigate in realistic 3D environments using natural language instructions. Existing VLN methods suffer from training on small-scale environments or unreasonable path-instruction datasets, limiting the generalization to unseen environments. There are massive house tour videos on YouTube, providing abundant real navigation experiences and layout information. However, these videos have not been explored for VLN before. In this paper, we propose to learn an agent from these videos by creating a large-scale dataset which comprises reasonable path-instruction pairs from house tour videos and pre-training the agent on it. To achieve this, we have to tackle the challenges of automatically constructing path-instruction pairs and exploiting real layout knowledge from raw and unlabeled videos. To address these, we first leverage an entropy-based method to construct the nodes of a path trajectory. Then, we propose an action-aware generator for generating instructions from unlabeled trajectories. Last, we devise a trajectory judgment pretext task to encourage the agent to mine the layout knowledge. Experimental results show that our method achieves state-of-the-art performance on two popular benchmarks (R2R and REVERIE). Code is available at https://github.com/JeremyLinky/YouTube-VLN","url_abs":"https://arxiv.org/abs/2307.11984v1","url_pdf":"https://arxiv.org/pdf/2307.11984v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-vision-and-language-navigation-from","repo_url":"https://github.com/jeremylinky/youtube-vln","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.11984","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.11984"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jeremylinky/youtube-vln","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":12,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":19,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"vilbert/vilbert.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/vilbert/vilbert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"c58840db809196b1","entry":"cached_path","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"vilbert/file_utils.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/vilbert/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c58840db809196b1"}},{"code_sha256_prefix":"e325858dc74dac65","entry":"filename_to_url","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"vilbert/file_utils.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/vilbert/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e325858dc74dac65"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"vilbert/vilbert.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/vilbert/vilbert.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"52c9873a3537fbf4","entry":"get_device","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/utils_init.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/utils_init.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52c9873a3537fbf4"}},{"code_sha256_prefix":"1efda40c7463ba10","entry":"get_logger","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/misc.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1efda40c7463ba10"}},{"code_sha256_prefix":"49aa48edebd62583","entry":"get_mask_options","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/utils_init.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/utils_init.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49aa48edebd62583"}},{"code_sha256_prefix":"33d4ad7b1fc7a2e0","entry":"get_model_input","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/utils_init.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/utils_init.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33d4ad7b1fc7a2e0"}},{"code_sha256_prefix":"1b9ab4ad9160ad73","entry":"get_output_dir","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/misc.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b9ab4ad9160ad73"}},{"code_sha256_prefix":"60fe7348f108a789","entry":"get_path","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/dataset/dataset_init.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/dataset/dataset_init.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60fe7348f108a789"}},{"code_sha256_prefix":"67de8cef340cc324","entry":"get_world_size","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/distributed.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67de8cef340cc324"}},{"code_sha256_prefix":"45e8b849b6a9b735","entry":"is_default_gpu","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/misc.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45e8b849b6a9b735"}},{"code_sha256_prefix":"6369d66e36fc6a2f","entry":"load_json_data","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/dataset/common.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/dataset/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6369d66e36fc6a2f"}},{"code_sha256_prefix":"b3d386bd340fd693","entry":"pad_packed","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/dataset/common.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/dataset/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3d386bd340fd693"}},{"code_sha256_prefix":"af64ec220e8bcdbc","entry":"url_to_filename","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"vilbert/file_utils.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/vilbert/file_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af64ec220e8bcdbc"}},{"code_sha256_prefix":"8d57e6390318ac16","entry":"get_rank","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/distributed.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d57e6390318ac16"}},{"code_sha256_prefix":"a835b42c7e48e4dd","entry":"get_testset_path","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/dataset/dataset_init.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/dataset/dataset_init.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a835b42c7e48e4dd"}},{"code_sha256_prefix":"6e80a4dbae93a9dc","entry":"load_nav_graphs","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"utils/dataset/common.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/utils/dataset/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e80a4dbae93a9dc"}},{"code_sha256_prefix":"baa5766f4566aafc","entry":"load_tf_weights_in_bert","repo":"jeremylinky/youtube-vln","repo_kind":"official","path":"vilbert/vilbert.py","file_url":"https://github.com/jeremylinky/youtube-vln/blob/HEAD/vilbert/vilbert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"baa5766f4566aafc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}