{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/3d-llm-injecting-the-3d-world-into-large","title":"3D-LLM: Injecting the 3D World into Large Language Models","arxiv_id":"2307.12981","date":"2023-07-24","proceeding":"NeurIPS 2023 11","authors":["Yining Hong","Haoyu Zhen","Peihao Chen","Shuhong Zheng","Yilun Du","Zhenfang Chen","Chuang Gan"],"abstract":"Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.","url_abs":"https://arxiv.org/abs/2307.12981v1","url_pdf":"https://arxiv.org/pdf/2307.12981v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"3d-llm-injecting-the-3d-world-into-large","repo_url":"https://github.com/Pointcept/GPT4Point","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"3d-llm-injecting-the-3d-world-into-large","repo_url":"https://github.com/Yui010206/CREMA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"3d-llm-injecting-the-3d-world-into-large","repo_url":"https://github.com/openrobotlab/pointllm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"3d-llm-injecting-the-3d-world-into-large","repo_url":"https://github.com/qizekun/ShapeLLM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"3d-llm-injecting-the-3d-world-into-large","repo_url":"https://github.com/umass-foundation-model/3d-llm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-object-captioning","task_name":"3D Object Captioning"},{"task_slug":"3d-question-answering-3d-qa","task_name":"3D Question Answering (3D-QA)"},{"task_slug":"dense-captioning","task_name":"Dense Captioning"},{"task_slug":"generative-3d-object-classification","task_name":"Generative 3D Object Classification"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-object-captioning-on-objaverse-1","task":"3D Object Captioning","dataset":"Objaverse","model":"3D-LLM","rank_in_archive_order":6,"of":6,"metrics":{" Sentence-BERT":"44.48","Correctness":"1.77","GPT-4":"33.42","Hallucination":"1.16","Precision":"60.39","SimCSE":"43.68"},"uses_additional_data":false},{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"3D-LLM (flamingo)","rank_in_archive_order":8,"of":18,"metrics":{"BLEU-1":"32.6","BLEU-4":"8.4","CIDEr":"65.6","Exact Match":"23.2","METEOR":"13.5","ROUGE":"34.8"},"uses_additional_data":false},{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"3D-LLM (BLIP2-flant5)","rank_in_archive_order":14,"of":18,"metrics":{"BLEU-1":"38.3","BLEU-4":"11.6","CIDEr":"69.6","Exact Match":"19.1","METEOR":"14.9","ROUGE":"35.3"},"uses_additional_data":false},{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"3D-LLM (BLIP2-opt)","rank_in_archive_order":15,"of":18,"metrics":{"BLEU-1":"37.3","BLEU-4":"10.7","CIDEr":"67.1","Exact Match":"19.1","METEOR":"14.3","ROUGE":"34.5"},"uses_additional_data":false},{"leaderboard":"/sota/generative-3d-object-classification-on-1","task":"Generative 3D Object Classification","dataset":"Objaverse","model":"3D-LLM","rank_in_archive_order":6,"of":7,"metrics":{"Objaverse (Average)":"45.25","Objaverse (C)":"41.50","Objaverse (I)":"49.00"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.12981","atlas_url":"https://app.syntology.ai/?focus=2307.12981","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.12981"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openrobotlab/pointllm","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Yui010206/CREMA","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/umass-foundation-model/3d-llm","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qizekun/ShapeLLM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Pointcept/GPT4Point","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_violates":1,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"listed":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"80dde39da1ba3708","entry":"compress","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLanguage_data/ChatCaptioner_based/quetion_model.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLanguage_data/ChatCaptioner_based/quetion_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"80dde39da1ba3708"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLLM_BLIP2-base/lavis/models/vit.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLLM_BLIP2-base/lavis/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"4e436e13094b20cb","entry":"processDigitArticle","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLLM_BLIP2-base/calculate_scores/calculate_score_3dmvvqa.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLLM_BLIP2-base/calculate_scores/calculate_score_3dmvvqa.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"4e436e13094b20cb"}},{"code_sha256_prefix":"35e84a8da1d4f427","entry":"processPunctuation","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLLM_BLIP2-base/calculate_scores/calculate_score_3dmvvqa.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLLM_BLIP2-base/calculate_scores/calculate_score_3dmvvqa.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"35e84a8da1d4f427"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLLM_BLIP2-base/lavis/models/base_model.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLLM_BLIP2-base/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLLM_BLIP2-base/lavis/models/base_model.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLLM_BLIP2-base/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"a2b79de775636d17","entry":"decompress","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLanguage_data/ChatCaptioner_based/quetion_model.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLanguage_data/ChatCaptioner_based/quetion_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a2b79de775636d17"}},{"code_sha256_prefix":"fe66d0280e436155","entry":"get_gpu_memory","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLanguage_data/ChatCaptioner_based/quetion_model.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLanguage_data/ChatCaptioner_based/quetion_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"fe66d0280e436155"}},{"code_sha256_prefix":"a1ff41c753bffe5d","entry":"get_lemma","repo":"umass-foundation-model/3d-llm","repo_kind":"listed","path":"3DLLM_BLIP2-base/calculate_scores/calculate_score_3dmvvqa.py","file_url":"https://github.com/umass-foundation-model/3d-llm/blob/HEAD/3DLLM_BLIP2-base/calculate_scores/calculate_score_3dmvvqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a1ff41c753bffe5d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}