{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-learning-a-generalist-model-for","title":"Towards Learning a Generalist Model for Embodied Navigation","arxiv_id":"2312.02010","date":"2023-12-04","proceeding":"CVPR 2024 1","authors":["Duo Zheng","Shijia Huang","Lin Zhao","Yiwu Zhong","LiWei Wang"],"abstract":"Building a generalist agent that can interact with the world is the intriguing target of AI systems, thus spurring the research for embodied navigation, where an agent is required to navigate according to instructions or respond to queries. Despite the major progress attained, previous works primarily focus on task-specific agents and lack generalizability to unseen scenarios. Recently, LLMs have presented remarkable capabilities across various fields, and provided a promising opportunity for embodied navigation. Drawing on this, we propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. The schema-based instruction flexibly casts various tasks into generation problems, thereby unifying a wide range of tasks. This approach allows us to integrate diverse data sources from various datasets into the training, equipping NaviLLM with a wide range of capabilities required by embodied navigation. We conduct extensive experiments to evaluate the performance and generalizability of our model. The experimental results demonstrate that our unified model achieves state-of-the-art performance on CVDN, SOON, and ScanQA. Specifically, it surpasses the previous stats-of-the-art method by a significant margin of 29% in goal progress on CVDN. Moreover, our model also demonstrates strong generalizability and presents impressive results on unseen tasks, e.g., embodied question answering and 3D captioning.","url_abs":"https://arxiv.org/abs/2312.02010v3","url_pdf":"https://arxiv.org/pdf/2312.02010v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-learning-a-generalist-model-for","repo_url":"https://github.com/lavi-lab/navillm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"towards-learning-a-generalist-model-for","repo_url":"https://github.com/zd11024/NaviLLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-question-answering-3d-qa","task_name":"3D Question Answering (3D-QA)"},{"task_slug":"embodied-question-answering","task_name":"Embodied Question Answering"},{"task_slug":"navigate","task_name":"Navigate"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-navigation","task_name":"Visual Navigation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"NaviLLM","rank_in_archive_order":5,"of":18,"metrics":{"BLEU-1":"39.73","BLEU-4":"13.90","CIDEr":"80.77","Exact Match":"26.27","METEOR":"16.56","ROUGE":"40.23"},"uses_additional_data":false},{"leaderboard":"/sota/visual-navigation-on-cooperative-vision-and-1","task":"Visual Navigation","dataset":"Cooperative Vision-and-Dialogue Navigation","model":"NaviLLM","rank_in_archive_order":1,"of":19,"metrics":{"dist_to_end_reduction":"7.90","spl":"0.09"},"uses_additional_data":false},{"leaderboard":"/sota/visual-navigation-on-room-to-room-1","task":"Visual Navigation","dataset":"R2R","model":"NaviLLM","rank_in_archive_order":4,"of":11,"metrics":{"spl":"0.60"},"uses_additional_data":false},{"leaderboard":"/sota/visual-navigation-on-soon-test","task":"Visual Navigation","dataset":"SOON Test","model":"NaviLLM","rank_in_archive_order":2,"of":6,"metrics":{"Nav-SPL":"26.26","SR":"35.04"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02010","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02010"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zd11024/NaviLLM","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lavi-lab/navillm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":5,"ran_violates":1,"ran_fixture":2,"ran_honours":1,"unverified":5},"by_repo_kind":{"official":{"samples":15,"ran":10,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/vln_bert.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/vln_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"dd2ac0ab9c587094","entry":"build_dataloader","repo":"lavi-lab/navillm","repo_kind":"official","path":"tasks/loaders.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/tasks/loaders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd2ac0ab9c587094"}},{"code_sha256_prefix":"c362aaed2935ceff","entry":"calc_position_distance","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/graph_utils.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/graph_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c362aaed2935ceff"}},{"code_sha256_prefix":"168e35f8ed1ad0ce","entry":"calculate_vp_rel_pos_fts","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/graph_utils.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/graph_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"168e35f8ed1ad0ce"}},{"code_sha256_prefix":"98a915bad0a2f94d","entry":"create_feature_db","repo":"lavi-lab/navillm","repo_kind":"official","path":"tasks/feature_db.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/tasks/feature_db.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98a915bad0a2f94d"}},{"code_sha256_prefix":"91e1742acf23aa57","entry":"extend_neg_masks","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/ops.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91e1742acf23aa57"}},{"code_sha256_prefix":"c2f0705ed8e6f7e6","entry":"gelu","repo":"zd11024/NaviLLM","repo_kind":"official","path":"models/vln_bert.py","file_url":"https://github.com/zd11024/NaviLLM/blob/HEAD/models/vln_bert.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2f0705ed8e6f7e6"}},{"code_sha256_prefix":"d9c76c6dc9976905","entry":"gen_seq_masks","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/ops.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9c76c6dc9976905"}},{"code_sha256_prefix":"dac1993f5ce8e12e","entry":"get_angle_fts","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/graph_utils.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/graph_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dac1993f5ce8e12e"}},{"code_sha256_prefix":"a581dc0bf92d2ee5","entry":"move_to_cuda","repo":"lavi-lab/navillm","repo_kind":"official","path":"tasks/loaders.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/tasks/loaders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a581dc0bf92d2ee5"}},{"code_sha256_prefix":"df2919fd1dad1d43","entry":"build_transformer","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/detr_transformer.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/detr_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df2919fd1dad1d43"}},{"code_sha256_prefix":"1a2aa37d868fb374","entry":"create_dataloaders","repo":"lavi-lab/navillm","repo_kind":"official","path":"tasks/loaders.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/tasks/loaders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a2aa37d868fb374"}},{"code_sha256_prefix":"bdc0b119cec4f508","entry":"create_transformer_encoder","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/ops.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdc0b119cec4f508"}},{"code_sha256_prefix":"7d0ae7f5c781116b","entry":"init_vis_config","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/nav_model.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/nav_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d0ae7f5c781116b"}},{"code_sha256_prefix":"6288ee4b92b46305","entry":"llama_model_in_debug_model","repo":"lavi-lab/navillm","repo_kind":"official","path":"models/vln_bert.py","file_url":"https://github.com/lavi-lab/navillm/blob/HEAD/models/vln_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6288ee4b92b46305"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}