{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/divscene-benchmarking-lvlms-for-object","title":"DivScene: Benchmarking LVLMs for Object Navigation with Diverse Scenes and Objects","arxiv_id":"2410.02730","date":"2024-10-03","proceeding":null,"authors":["Zhaowei Wang","Hongming Zhang","Tianqing Fang","Ye Tian","Yue Yang","Kaixin Ma","Xiaoman Pan","Yangqiu Song","Dong Yu"],"abstract":"Object navigation in unknown environments is crucial for deploying embodied agents in real-world applications. While we have witnessed huge progress due to large-scale scene datasets, faster simulators, and stronger models, previous studies mainly focus on limited scene types and target objects. In this paper, we study a new task of navigating to diverse target objects in a large number of scene types. To benchmark the problem, we present a large-scale scene dataset, DivScene, which contains 4,614 scenes across 81 different types. With the dataset, we build an end-to-end embodied agent, NatVLM, by fine-tuning a Large Vision Language Model (LVLM) through imitation learning. The LVLM is trained to take previous observations from the environment and generate the next actions. We also introduce CoT explanation traces of the action prediction for better performance when tuning LVLMs. Our extensive experiments find that we can build a performant LVLM-based agent through imitation learning on the shortest paths constructed by a BFS planner without any human supervision. Our agent achieves a success rate that surpasses GPT-4o by over 20%. Meanwhile, we carry out various analyses showing the generalization ability of our agent. Our code and data are available at https://github.com/zhaowei-wang-nlp/DivScene.","url_abs":"https://arxiv.org/abs/2410.02730v2","url_pdf":"https://arxiv.org/pdf/2410.02730v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"divscene-benchmarking-lvlms-for-object","repo_url":"https://github.com/zhaowei-wang-nlp/divscene","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.02730","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02730"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaowei-wang-nlp/divscene","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":15},"by_repo_kind":{"official":{"samples":15,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c398e46c3e1c3747","entry":"calculate_path_length","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_metric.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c398e46c3e1c3747"}},{"code_sha256_prefix":"4259267f7d4a7e91","entry":"convert_pair_to_str","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"reformat_data/diff_utils.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/reformat_data/diff_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4259267f7d4a7e91"}},{"code_sha256_prefix":"005fdbd14f429fc0","entry":"extract_object_type","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"reformat_data/diff_utils.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/reformat_data/diff_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"005fdbd14f429fc0"}},{"code_sha256_prefix":"f3f346ac0d3677e3","entry":"format_args","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/utils.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3f346ac0d3677e3"}},{"code_sha256_prefix":"9f5266890fdc41a6","entry":"get_cur_diff","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"reformat_data/diff_utils.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/reformat_data/diff_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f5266890fdc41a6"}},{"code_sha256_prefix":"346ba3a0aa9a8e55","entry":"get_cur_diff","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"reformat_data/utils.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/reformat_data/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"346ba3a0aa9a8e55"}},{"code_sha256_prefix":"44465377eaacf505","entry":"get_shortest_path_to_point","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_online_client.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_online_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44465377eaacf505"}},{"code_sha256_prefix":"b297d01d782e10b4","entry":"get_type_dict","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_gpt4o.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b297d01d782e10b4"}},{"code_sha256_prefix":"4650a677e33f265a","entry":"init_logger","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/utils.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4650a677e33f265a"}},{"code_sha256_prefix":"b43f513176e6bbfe","entry":"map_action_to_id","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_gpt4o.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b43f513176e6bbfe"}},{"code_sha256_prefix":"e04797a75615b923","entry":"map_action_to_id","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_server.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e04797a75615b923"}},{"code_sha256_prefix":"b6552010f6d8a287","entry":"parse_action","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_gpt4o.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6552010f6d8a287"}},{"code_sha256_prefix":"7a16f1f9f950f857","entry":"parse_action","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_online_client.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_online_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a16f1f9f950f857"}},{"code_sha256_prefix":"a24d87222eb1a622","entry":"parse_action","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/hf_idefics2_server.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/hf_idefics2_server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a24d87222eb1a622"}},{"code_sha256_prefix":"a7b36e290c6f5fc1","entry":"revise_mnli_models","repo":"zhaowei-wang-nlp/divscene","repo_kind":"official","path":"agent_inference/utils.py","file_url":"https://github.com/zhaowei-wang-nlp/divscene/blob/HEAD/agent_inference/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7b36e290c6f5fc1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}