{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ll3da-visual-interactive-instruction-tuning","title":"LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning","arxiv_id":"2311.18651","date":"2023-11-30","proceeding":null,"authors":["Sijin Chen","Xin Chen","Chi Zhang","Mingsheng Li","Gang Yu","Hao Fei","Hongyuan Zhu","Jiayuan Fan","Tao Chen"],"abstract":"Recent advances in Large Multimodal Models (LMM) have made it possible for various applications in human-machine interactions. However, developing LMMs that can comprehend, reason, and plan in complex and diverse 3D environments remains a challenging topic, especially considering the demand for understanding permutation-invariant point cloud 3D representations of the 3D scene. Existing works seek help from multi-view images, and project 2D features to 3D space as 3D scene representations. This, however, leads to huge computational overhead and performance degradation. In this paper, we present LL3DA, a Large Language 3D Assistant that takes point cloud as direct input and respond to both textual-instructions and visual-prompts. This help LMMs better comprehend human interactions and further help to remove the ambiguities in cluttered 3D scenes. Experiments show that LL3DA achieves remarkable results, and surpasses various 3D vision-language models on both 3D Dense Captioning and 3D Question Answering.","url_abs":"https://arxiv.org/abs/2311.18651v1","url_pdf":"https://arxiv.org/pdf/2311.18651v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ll3da-visual-interactive-instruction-tuning","repo_url":"https://github.com/open3da/ll3da","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-dense-captioning","task_name":"3D dense captioning"},{"task_slug":"dense-captioning","task_name":"Dense Captioning"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2311.18651","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.18651"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/open3da/ll3da","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f4eafced0f46835f","entry":"adjust_learning_rate","repo":"open3da/ll3da","repo_kind":"official","path":"engine.py","file_url":"https://github.com/open3da/ll3da/blob/HEAD/engine.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4eafced0f46835f"}},{"code_sha256_prefix":"4ae1b9a3a559ac1e","entry":"beam_search_decode","repo":"open3da/ll3da","repo_kind":"official","path":"models/ll3da/generation_utils.py","file_url":"https://github.com/open3da/ll3da/blob/HEAD/models/ll3da/generation_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ae1b9a3a559ac1e"}},{"code_sha256_prefix":"289df3d353cb834e","entry":"compute_learning_rate","repo":"open3da/ll3da","repo_kind":"official","path":"engine.py","file_url":"https://github.com/open3da/ll3da/blob/HEAD/engine.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"289df3d353cb834e"}},{"code_sha256_prefix":"ffbc06852635802f","entry":"generation","repo":"open3da/ll3da","repo_kind":"official","path":"models/ll3da/generation_utils.py","file_url":"https://github.com/open3da/ll3da/blob/HEAD/models/ll3da/generation_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffbc06852635802f"}},{"code_sha256_prefix":"775ed3f78bf348a5","entry":"greedy_decode","repo":"open3da/ll3da","repo_kind":"official","path":"models/ll3da/generation_utils.py","file_url":"https://github.com/open3da/ll3da/blob/HEAD/models/ll3da/generation_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"775ed3f78bf348a5"}},{"code_sha256_prefix":"1635df9b25a2d8d1","entry":"proposal_dimension_select","repo":"open3da/ll3da","repo_kind":"official","path":"models/ll3da/captioner.py","file_url":"https://github.com/open3da/ll3da/blob/HEAD/models/ll3da/captioner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1635df9b25a2d8d1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}