{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/m3dbench-let-s-instruct-large-models-with","title":"M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts","arxiv_id":"2312.10763","date":"2023-12-17","proceeding":null,"authors":["Mingsheng Li","Xin Chen","Chi Zhang","Sijin Chen","Hongyuan Zhu","Fukun Yin","Gang Yu","Tao Chen"],"abstract":"Recently, 3D understanding has become popular to facilitate autonomous agents to perform further decisionmaking. However, existing 3D datasets and methods are often limited to specific tasks. On the other hand, recent progress in Large Language Models (LLMs) and Multimodal Language Models (MLMs) have demonstrated exceptional general language and imagery tasking performance. Therefore, it is interesting to unlock MLM's potential to be 3D generalist for wider tasks. However, current MLMs' research has been less focused on 3D tasks due to a lack of large-scale 3D instruction-following datasets. In this work, we introduce a comprehensive 3D instructionfollowing dataset called M3DBench, which possesses the following characteristics: 1) It supports general multimodal instructions interleaved with text, images, 3D objects, and other visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels, covering a variety of fundamental abilities in real-world 3D environments. 3) It is a large-scale 3D instruction-following dataset with over 320k instruction-response pairs. Furthermore, we establish a new benchmark for assessing the performance of large models in understanding multi-modal 3D prompts. Extensive experiments demonstrate the effectiveness of our dataset and baseline, supporting general 3D-centric tasks, which can inspire future research.","url_abs":"https://arxiv.org/abs/2312.10763v1","url_pdf":"https://arxiv.org/pdf/2312.10763v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"m3dbench-let-s-instruct-large-models-with","repo_url":"https://github.com/OpenM3D/M3DBench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"m3dbench-let-s-instruct-large-models-with","repo_url":"https://github.com/unimodal4reasoning/geox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.10763","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.10763"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/unimodal4reasoning/geox","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OpenM3D/M3DBench","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"91a15c2e2e44fabc","entry":"adjust_learning_rate","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"engine.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91a15c2e2e44fabc"}},{"code_sha256_prefix":"4ae1b9a3a559ac1e","entry":"beam_search_decode","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"models/llm_decoder/generation_utils.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/models/llm_decoder/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ae1b9a3a559ac1e"}},{"code_sha256_prefix":"f46bfbc2eff05113","entry":"compute_learning_rate","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"engine.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f46bfbc2eff05113"}},{"code_sha256_prefix":"ffbc06852635802f","entry":"generation","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"models/llm_decoder/generation_utils.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/models/llm_decoder/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffbc06852635802f"}},{"code_sha256_prefix":"891b8ebab395921f","entry":"get_clones","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"models/transformer_encoder/helpers.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/models/transformer_encoder/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"891b8ebab395921f"}},{"code_sha256_prefix":"775ed3f78bf348a5","entry":"greedy_decode","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"models/llm_decoder/generation_utils.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/models/llm_decoder/generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"775ed3f78bf348a5"}},{"code_sha256_prefix":"5712c23e0574f717","entry":"load_all","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"eval_utils/util.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/eval_utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5712c23e0574f717"}},{"code_sha256_prefix":"e2cb1d5c8f9ee592","entry":"prepare_corpus","repo":"OpenM3D/M3DBench","repo_kind":"official","path":"eval_utils/util.py","file_url":"https://github.com/OpenM3D/M3DBench/blob/HEAD/eval_utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2cb1d5c8f9ee592"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}