{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gpt4point-a-unified-framework-for-point","title":"GPT4Point: A Unified Framework for Point-Language Understanding and Generation","arxiv_id":"2312.02980","date":"2023-12-05","proceeding":"CVPR 2024 1","authors":["Zhangyang Qi","Ye Fang","Zeyi Sun","Xiaoyang Wu","Tong Wu","Jiaqi Wang","Dahua Lin","Hengshuang Zhao"],"abstract":"Multimodal Large Language Models (MLLMs) have excelled in 2D image-text comprehension and image generation, but their understanding of the 3D world is notably deficient, limiting progress in 3D language understanding and generation. To solve this problem, we introduce GPT4Point, an innovative groundbreaking point-language multimodal model designed specifically for unified 3D object understanding and generation within the MLLM framework. GPT4Point as a powerful 3D MLLM seamlessly can execute a variety of point-text reference tasks such as point-cloud captioning and Q&A. Additionally, GPT4Point is equipped with advanced capabilities for controllable 3D generation, it can get high-quality results through a low-quality point-text feature maintaining the geometric shapes and colors. To support the expansive needs of 3D object-text pairs, we develop Pyramid-XL, a point-language dataset annotation engine. It constructs a large-scale database over 1M objects of varied text granularity levels from the Objaverse-XL dataset, essential for training GPT4Point. A comprehensive benchmark has been proposed to evaluate 3D point-language understanding capabilities. In extensive evaluations, GPT4Point has demonstrated superior performance in understanding and generation.","url_abs":"https://arxiv.org/abs/2312.02980v1","url_pdf":"https://arxiv.org/pdf/2312.02980v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gpt4point-a-unified-framework-for-point","repo_url":"https://github.com/Pointcept/GPT4Point","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-generation","task_name":"3D Generation"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2312.02980","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02980"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Pointcept/GPT4Point","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":2,"unverified":2},"by_repo_kind":{"listed":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"Pointcept/GPT4Point","repo_kind":"listed","path":"lavis/models/gpt4point_models/gpt4point.py","file_url":"https://github.com/Pointcept/GPT4Point/blob/HEAD/lavis/models/gpt4point_models/gpt4point.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"7b9471aa6c1ba723","entry":"load_npz_file","repo":"Pointcept/GPT4Point","repo_kind":"listed","path":"Objaverse-xl_Download/shap-e/npz2ply.py","file_url":"https://github.com/Pointcept/GPT4Point/blob/HEAD/Objaverse-xl_Download/shap-e/npz2ply.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b9471aa6c1ba723"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"Pointcept/GPT4Point","repo_kind":"listed","path":"lavis/models/base_model.py","file_url":"https://github.com/Pointcept/GPT4Point/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"Pointcept/GPT4Point","repo_kind":"listed","path":"lavis/models/base_model.py","file_url":"https://github.com/Pointcept/GPT4Point/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"a7740951e95de5ee","entry":"get_example_objects","repo":"Pointcept/GPT4Point","repo_kind":"listed","path":"Objaverse-xl_Download/download.py","file_url":"https://github.com/Pointcept/GPT4Point/blob/HEAD/Objaverse-xl_Download/download.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7740951e95de5ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}