{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sim-to-real-transfer-via-3d-feature-fields","title":"Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation","arxiv_id":"2406.09798","date":"2024-06-14","proceeding":null,"authors":["Zihan Wang","Xiangyang Li","Jiahao Yang","Yeqi Liu","Shuqiang Jiang"],"abstract":"Vision-and-language navigation (VLN) enables the agent to navigate to a remote location in 3D environments following the natural language instruction. In this field, the agent is usually trained and evaluated in the navigation simulators, lacking effective approaches for sim-to-real transfer. The VLN agents with only a monocular camera exhibit extremely limited performance, while the mainstream VLN models trained with panoramic observation, perform better but are difficult to deploy on most monocular robots. For this case, we propose a sim-to-real transfer approach to endow the monocular robots with panoramic traversability perception and panoramic semantic understanding, thus smoothly transferring the high-performance panoramic VLN models to the common monocular robots. In this work, the semantic traversable map is proposed to predict agent-centric navigable waypoints, and the novel view representations of these navigable waypoints are predicted through the 3D feature fields. These methods broaden the limited field of view of the monocular robots and significantly improve navigation performance in the real world. Our VLN system outperforms previous SOTA monocular VLN methods in R2R-CE and RxR-CE benchmarks within the simulation environments and is also validated in real-world environments, providing a practical and high-performance solution for real-world VLN.","url_abs":"https://arxiv.org/abs/2406.09798v3","url_pdf":"https://arxiv.org/pdf/2406.09798v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sim-to-real-transfer-via-3d-feature-fields","repo_url":"https://github.com/MrZihan/Sim2Real-VLN-3DFF","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.09798","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.09798"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MrZihan/Sim2Real-VLN-3DFF","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"e3de2e386766be00","entry":"static_to_grid","repo":"MrZihan/Sim2Real-VLN-3DFF","repo_kind":"official","path":"VLN_3DFF/habitat_extensions/maps.py","file_url":"https://github.com/MrZihan/Sim2Real-VLN-3DFF/blob/HEAD/VLN_3DFF/habitat_extensions/maps.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3de2e386766be00"}},{"code_sha256_prefix":"f474c29ed36d8065","entry":"collate_fn","repo":"MrZihan/Sim2Real-VLN-3DFF","repo_kind":"official","path":"VLN_3DFF/vlnce_baselines/dagger_trainer.py","file_url":"https://github.com/MrZihan/Sim2Real-VLN-3DFF/blob/HEAD/VLN_3DFF/vlnce_baselines/dagger_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f474c29ed36d8065"}},{"code_sha256_prefix":"3559a5421dd17ce5","entry":"colorize_topdown_map","repo":"MrZihan/Sim2Real-VLN-3DFF","repo_kind":"official","path":"VLN_3DFF/habitat_extensions/maps.py","file_url":"https://github.com/MrZihan/Sim2Real-VLN-3DFF/blob/HEAD/VLN_3DFF/habitat_extensions/maps.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3559a5421dd17ce5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}