{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gpt4scene-understand-3d-scenes-from-videos","title":"GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models","arxiv_id":"2501.01428","date":"2025-01-02","proceeding":null,"authors":["Zhangyang Qi","Zhixiong Zhang","Ye Fang","Jiaqi Wang","Hengshuang Zhao"],"abstract":"In recent years, 2D Vision-Language Models (VLMs) have made significant strides in image-text understanding tasks. However, their performance in 3D spatial comprehension, which is critical for embodied intelligence, remains limited. Recent advances have leveraged 3D point clouds and multi-view images as inputs, yielding promising results. However, we propose exploring a purely vision-based solution inspired by human perception, which merely relies on visual cues for 3D spatial understanding. This paper empirically investigates the limitations of VLMs in 3D spatial knowledge, revealing that their primary shortcoming lies in the lack of global-local correspondence between the scene and individual frames. To address this, we introduce GPT4Scene, a novel visual prompting paradigm in VLM training and inference that helps build the global-local relationship, significantly improving the 3D spatial understanding of indoor scenes. Specifically, GPT4Scene constructs a 3D Bird's Eye View (BEV) image from the video and marks consistent object IDs across both frames and the BEV image. The model then inputs the concatenated BEV image and video frames with markers. In zero-shot evaluations, GPT4Scene improves performance over closed-source VLMs like GPT-4o. Additionally, we prepare a processed video dataset consisting of 165K text annotation to fine-tune open-source VLMs, achieving state-of-the-art performance on all 3D understanding tasks. Surprisingly, after training with the GPT4Scene paradigm, VLMs consistently improve during inference, even without visual prompting and BEV image as explicit correspondence. It demonstrates that the proposed paradigm helps VLMs develop an intrinsic ability to understand 3D scenes, which paves the way for a noninvasive approach to extending pre-trained VLMs for 3D scene understanding.","url_abs":"https://arxiv.org/abs/2501.01428v3","url_pdf":"https://arxiv.org/pdf/2501.01428v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gpt4scene-understand-3d-scenes-from-videos","repo_url":"https://github.com/Qi-Zhangyang/GPT4Scene","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"visual-prompting","task_name":"Visual Prompting"},{"task_slug":"text-annotation","task_name":"text annotation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2501.01428","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.01428"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Qi-Zhangyang/GPT4Scene","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d7bf7ad0e60457f","entry":"create_sampler","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"evaluate/val_dataset.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/evaluate/val_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d7bf7ad0e60457f"}},{"code_sha256_prefix":"5064009b43a00726","entry":"add_step_prompts","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"VLN-Ego-making/VLN-Ego/r2r_rxr_dataset_convert_sft.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/VLN-Ego-making/VLN-Ego/r2r_rxr_dataset_convert_sft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5064009b43a00726"}},{"code_sha256_prefix":"7173826fa5a3b3ec","entry":"bulid_message","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"evaluate/qwen2_vl.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/evaluate/qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7173826fa5a3b3ec"}},{"code_sha256_prefix":"d1cc4570ba8bbd09","entry":"create_dataset","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"evaluate/val_dataset.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/evaluate/val_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d1cc4570ba8bbd09"}},{"code_sha256_prefix":"c11a23c56c87bd2d","entry":"create_loader","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"evaluate/val_dataset.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/evaluate/val_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c11a23c56c87bd2d"}},{"code_sha256_prefix":"13da697eeaf18b88","entry":"create_navigation_data","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"VLN-Ego-making/VLN-Ego/r2r_rxr_dataset_convert_sft.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/VLN-Ego-making/VLN-Ego/r2r_rxr_dataset_convert_sft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13da697eeaf18b88"}},{"code_sha256_prefix":"10db9373c4cbf700","entry":"get_step_images","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"VLN-Ego-making/VLN-Ego/r2r_rxr_dataset_convert_sft.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/VLN-Ego-making/VLN-Ego/r2r_rxr_dataset_convert_sft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10db9373c4cbf700"}},{"code_sha256_prefix":"174c802b9d2bb222","entry":"preprocess_image_nearest","repo":"Qi-Zhangyang/GPT4Scene","repo_kind":"official","path":"evaluate/qwen2_vl.py","file_url":"https://github.com/Qi-Zhangyang/GPT4Scene/blob/HEAD/evaluate/qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"174c802b9d2bb222"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}