{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/human-aware-vision-and-language-navigation","title":"Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions","arxiv_id":"2406.19236","date":"2024-06-27","proceeding":null,"authors":["Heng Li","Minghan Li","Zhi-Qi Cheng","Yifei Dong","Yuxuan Zhou","Jun-Yan He","Qi Dai","Teruko Mitamura","Alexander G. Hauptmann"],"abstract":"Vision-and-Language Navigation (VLN) aims to develop embodied agents that navigate based on human instructions. However, current VLN frameworks often rely on static environments and optimal expert supervision, limiting their real-world applicability. To address this, we introduce Human-Aware Vision-and-Language Navigation (HA-VLN), extending traditional VLN by incorporating dynamic human activities and relaxing key assumptions. We propose the Human-Aware 3D (HA3D) simulator, which combines dynamic human activities with the Matterport3D dataset, and the Human-Aware Room-to-Room (HA-R2R) dataset, extending R2R with human activity descriptions. To tackle HA-VLN challenges, we present the Expert-Supervised Cross-Modal (VLN-CM) and Non-Expert-Supervised Decision Transformer (VLN-DT) agents, utilizing cross-modal fusion and diverse training strategies for effective navigation in dynamic human environments. A comprehensive evaluation, including metrics considering human activities, and systematic analysis of HA-VLN's unique challenges, underscores the need for further research to enhance HA-VLN agents' real-world robustness and adaptability. Ultimately, this work provides benchmarks and insights for future research on embodied AI and Sim2Real transfer, paving the way for more realistic and applicable VLN systems in human-populated environments.","url_abs":"https://arxiv.org/abs/2406.19236v3","url_pdf":"https://arxiv.org/pdf/2406.19236v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"human-aware-vision-and-language-navigation","repo_url":"https://github.com/lpercc/ha3d_simulator","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"vision-and-language-navigation","task_name":"Vision and Language Navigation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.19236","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.19236"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/peteanderson80/Matterport3DSimulator","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lpercc/ha3d_simulator","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1},"found_in_text":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"3bf5245a56c56be1","entry":"AttnDecoderLSTM","repo":"peteanderson80/Matterport3DSimulator","repo_kind":"found_in_text","path":"tasks/R2R/model.py","file_url":"https://github.com/peteanderson80/Matterport3DSimulator/blob/HEAD/tasks/R2R/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3bf5245a56c56be1"}},{"code_sha256_prefix":"fcb4c7f24c6fdfb4","entry":"SoftDotAttention","repo":"peteanderson80/Matterport3DSimulator","repo_kind":"found_in_text","path":"tasks/R2R/model.py","file_url":"https://github.com/peteanderson80/Matterport3DSimulator/blob/HEAD/tasks/R2R/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fcb4c7f24c6fdfb4"}},{"code_sha256_prefix":"da343e316b1e1233","entry":"compute","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"human-viewpoint_annotation/GUI.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/human-viewpoint_annotation/GUI.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da343e316b1e1233"}},{"code_sha256_prefix":"c0a6c9ca8a2e54b8","entry":"getRotation","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"src/render/renderer.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/src/render/renderer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0a6c9ca8a2e54b8"}},{"code_sha256_prefix":"02200133f4a15ecf","entry":"getRotation","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"src/render/rendermdm.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/src/render/rendermdm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02200133f4a15ecf"}},{"code_sha256_prefix":"1f3d937a2c0d07f8","entry":"locationTypeDic","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"HA3DSim.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/HA3DSim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f3d937a2c0d07f8"}},{"code_sha256_prefix":"8d66151ac83216bc","entry":"navigableLocationsTypeDic","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"HA3DSim.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/HA3DSim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d66151ac83216bc"}},{"code_sha256_prefix":"9c4b4864491fd086","entry":"renderFrames","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"src/render/rendermdm.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/src/render/rendermdm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c4b4864491fd086"}},{"code_sha256_prefix":"f95217d6e5b8f7e6","entry":"stateToDic","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"HA3DSim.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/HA3DSim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f95217d6e5b8f7e6"}},{"code_sha256_prefix":"69b68e12100d16b4","entry":"computeRel","repo":"lpercc/ha3d_simulator","repo_kind":"official","path":"src/render/rendermdm.py","file_url":"https://github.com/lpercc/ha3d_simulator/blob/HEAD/src/render/rendermdm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69b68e12100d16b4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}