{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-from-videos-for-3d-world-enhancing","title":"Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors","arxiv_id":"2505.24625","date":"2025-05-30","proceeding":null,"authors":["Duo Zheng","Shijia Huang","Yanyang Li","LiWei Wang"],"abstract":"Previous research has investigated the application of Multimodal Large Language Models (MLLMs) in understanding 3D scenes by interpreting them as videos. These approaches generally depend on comprehensive 3D data inputs, such as point clouds or reconstructed Bird's-Eye View (BEV) maps. In our research, we advance this field by enhancing the capability of MLLMs to understand and reason in 3D spaces directly from video data, without the need for additional 3D input. We propose a novel and efficient method, the Video-3D Geometry Large Language Model (VG LLM). Our approach employs a 3D visual geometry encoder that extracts 3D prior information from video sequences. This information is integrated with visual tokens and fed into the MLLM. Extensive experiments have shown that our method has achieved substantial improvements in various tasks related to 3D scene understanding and spatial reasoning, all directly learned from video sources. Impressively, our 4B model, which does not rely on explicit 3D data inputs, achieves competitive results compared to existing state-of-the-art methods, and even surpasses the Gemini-1.5-Pro in the VSI-Bench evaluations.","url_abs":"https://arxiv.org/abs/2505.24625v1","url_pdf":"https://arxiv.org/pdf/2505.24625v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-from-videos-for-3d-world-enhancing","repo_url":"https://github.com/LaVi-Lab/Video-3D-LLM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"3d-geometry","task_name":"3D geometry"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.24625","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.24625"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LaVi-Lab/Video-3D-LLM","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"ran":1,"unverified":8},"by_repo_kind":{"listed":{"samples":12,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b0a851cae92754ff","entry":"extract_patches","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/mm_utils.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a851cae92754ff"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/eval/model_vqa.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"69876e2beb68c07e","entry":"load_matrix_from_txt","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/video_utils.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/video_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69876e2beb68c07e"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/eval/model_vqa.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"3c8823948724e269","entry":"auto_pad_images","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/mm_utils.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c8823948724e269"}},{"code_sha256_prefix":"d0dc20ee33543a67","entry":"convert_from_uvd","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/video_utils.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/video_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d0dc20ee33543a67"}},{"code_sha256_prefix":"c32b636f13ed9ee5","entry":"convert_to_scalar","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"trl/core.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c32b636f13ed9ee5"}},{"code_sha256_prefix":"a51ae45a1f6c0b5b","entry":"flatten_dict","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"trl/core.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a51ae45a1f6c0b5b"}},{"code_sha256_prefix":"bdf0034b99719eac","entry":"resize_and_center_crop","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/mm_utils.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdf0034b99719eac"}},{"code_sha256_prefix":"b20705b0dfd77ffc","entry":"safe_load_tokenizer","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/conversation.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b20705b0dfd77ffc"}},{"code_sha256_prefix":"2c3da386578af970","entry":"top_k_top_p_filtering","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"trl/core.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c3da386578af970"}},{"code_sha256_prefix":"65d0d7acf1062983","entry":"unproject","repo":"LaVi-Lab/Video-3D-LLM","repo_kind":"listed","path":"llava/video_utils.py","file_url":"https://github.com/LaVi-Lab/Video-3D-LLM/blob/HEAD/llava/video_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65d0d7acf1062983"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}