{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chat-3d-v2-bridging-3d-scene-and-large","title":"Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers","arxiv_id":"2312.08168","date":"2023-12-13","proceeding":null,"authors":["Haifeng Huang","Yilun Chen","Zehan Wang","Rongjie Huang","Runsen Xu","Tai Wang","Luping Liu","Xize Cheng","Yang Zhao","Jiangmiao Pang","Zhou Zhao"],"abstract":"Recent advancements in 3D Large Language Models (LLMs) have demonstrated promising capabilities for 3D scene understanding. However, previous methods exhibit deficiencies in general referencing and grounding capabilities for intricate scene comprehension. In this paper, we introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Specifically, we decompose the input 3D scene into a set of object proposals, each assigned a unique identifier token, which enables efficient object referencing and grounding during user-assistant interactions. Given the scarcity of scene-language data, we model the scene embeddings as a sequence of explicit object-level embeddings, derived from semantic-rich 2D or 3D representations. By employing object identifiers, we transform diverse 3D scene-language tasks into a unified question-answering format, facilitating joint training without the need for additional task-specific heads. With minimal fine-tuning on all downstream tasks, our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.","url_abs":"https://arxiv.org/abs/2312.08168v4","url_pdf":"https://arxiv.org/pdf/2312.08168v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chat-3d-v2-bridging-3d-scene-and-large","repo_url":"https://github.com/chat-3d/chat-3d-v2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"chat-3d-v2-bridging-3d-scene-and-large","repo_url":"https://github.com/Chat-3D/Chat-3D","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-question-answering-3d-qa","task_name":"3D Question Answering (3D-QA)"},{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"object","task_name":"Object"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-question-answering-3d-qa-on-sqa3d","task":"3D Question Answering (3D-QA)","dataset":"SQA3D","model":"Chat-3D v2","rank_in_archive_order":3,"of":13,"metrics":{"Exact Match":"54.7"},"uses_additional_data":false},{"leaderboard":"/sota/3d-question-answering-3d-qa-on-sqa3d","task":"3D Question Answering (3D-QA)","dataset":"SQA3D","model":"ChatScene","rank_in_archive_order":4,"of":13,"metrics":{"Exact Match":"54.6"},"uses_additional_data":false},{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"ChatScene","rank_in_archive_order":10,"of":18,"metrics":{"BLEU-4":"14.3","CIDEr":"87.7","Exact Match":"21.6","METEOR":"18.0","ROUGE":"41.6"},"uses_additional_data":false},{"leaderboard":"/sota/3d-question-answering-3d-qa-on-scanqa-test-w","task":"3D Question Answering (3D-QA)","dataset":"ScanQA Test w/ objects","model":"Chat-3D v2","rank_in_archive_order":17,"of":18,"metrics":{"BLEU-1":"-","BLEU-4":"14.0","CIDEr":"87.6","Exact Match":"-","METEOR":"-","ROUGE":"-"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.08168","atlas_url":"https://app.syntology.ai/?focus=2312.08168","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.08168"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Chat-3D/Chat-3D","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chat-3d/chat-3d-v2","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":3,"ran":6,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1},"listed":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"chat-3d/chat-3d-v2","repo_kind":"official","path":"models/modeling_llama.py","file_url":"https://github.com/chat-3d/chat-3d-v2/blob/HEAD/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"chat-3d/chat-3d-v2","repo_kind":"official","path":"models/modeling_llama.py","file_url":"https://github.com/chat-3d/chat-3d-v2/blob/HEAD/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"9b4dff79d5e6102c","entry":"apply_rotary_pos_emb","repo":"Chat-3D/Chat-3D","repo_kind":"listed","path":"models/modeling_llama.py","file_url":"https://github.com/Chat-3D/Chat-3D/blob/HEAD/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b4dff79d5e6102c"}},{"code_sha256_prefix":"e0618e4feaada458","entry":"compute_mhsa","repo":"Chat-3D/Chat-3D","repo_kind":"listed","path":"models/transformer_vanilla/mhsa.py","file_url":"https://github.com/Chat-3D/Chat-3D/blob/HEAD/models/transformer_vanilla/mhsa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0618e4feaada458"}},{"code_sha256_prefix":"cda0cda020ca6dee","entry":"get_caption","repo":"chat-3d/chat-3d-v2","repo_kind":"official","path":"preprocess/prepare_scannet_region_caption_annos.py","file_url":"https://github.com/chat-3d/chat-3d-v2/blob/HEAD/preprocess/prepare_scannet_region_caption_annos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cda0cda020ca6dee"}},{"code_sha256_prefix":"891b8ebab395921f","entry":"get_clones","repo":"Chat-3D/Chat-3D","repo_kind":"listed","path":"models/helpers.py","file_url":"https://github.com/Chat-3D/Chat-3D/blob/HEAD/models/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"891b8ebab395921f"}},{"code_sha256_prefix":"3f8b2fc0609a5184","entry":"nclamp","repo":"chat-3d/chat-3d-v2","repo_kind":"official","path":"models/chat3d.py","file_url":"https://github.com/chat-3d/chat-3d-v2/blob/HEAD/models/chat3d.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f8b2fc0609a5184"}},{"code_sha256_prefix":"5ec87684b9345e0a","entry":"process_batch_data","repo":"Chat-3D/Chat-3D","repo_kind":"listed","path":"dataset/base_dataset.py","file_url":"https://github.com/Chat-3D/Chat-3D/blob/HEAD/dataset/base_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ec87684b9345e0a"}},{"code_sha256_prefix":"3895c5e4b8839cf6","entry":"recover_caption","repo":"chat-3d/chat-3d-v2","repo_kind":"official","path":"preprocess/prepare_scannet_region_caption_annos.py","file_url":"https://github.com/chat-3d/chat-3d-v2/blob/HEAD/preprocess/prepare_scannet_region_caption_annos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3895c5e4b8839cf6"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"Chat-3D/Chat-3D","repo_kind":"listed","path":"models/modeling_llama.py","file_url":"https://github.com/Chat-3D/Chat-3D/blob/HEAD/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"8ea16420707b36f4","entry":"update_caption","repo":"chat-3d/chat-3d-v2","repo_kind":"official","path":"preprocess/prepare_scannet_region_caption_annos.py","file_url":"https://github.com/chat-3d/chat-3d-v2/blob/HEAD/preprocess/prepare_scannet_region_caption_annos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ea16420707b36f4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}