{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/global-local-collaborative-inference-with-llm","title":"Global-Local Collaborative Inference with LLM for Lidar-Based Open-Vocabulary Detection","arxiv_id":"2407.08931","date":"2024-07-12","proceeding":null,"authors":["Xingyu Peng","Yan Bai","Chen Gao","Lirong Yang","Fei Xia","Beipeng Mu","Xiaofei Wang","Si Liu"],"abstract":"Open-Vocabulary Detection (OVD) is the task of detecting all interesting objects in a given scene without predefined object classes. Extensive work has been done to deal with the OVD for 2D RGB images, but the exploration of 3D OVD is still limited. Intuitively, lidar point clouds provide 3D information, both object level and scene level, to generate trustful detection results. However, previous lidar-based OVD methods only focus on the usage of object-level features, ignoring the essence of scene-level information. In this paper, we propose a Global-Local Collaborative Scheme (GLIS) for the lidar-based OVD task, which contains a local branch to generate object-level detection result and a global branch to obtain scene-level global feature. With the global-local information, a Large Language Model (LLM) is applied for chain-of-thought inference, and the detection result can be refined accordingly. We further propose Reflected Pseudo Labels Generation (RPLG) to generate high-quality pseudo labels for supervision and Background-Aware Object Localization (BAOL) to select precise object proposals. Extensive experiments on ScanNetV2 and SUN RGB-D demonstrate the superiority of our methods. Code is released at https://github.com/GradiusTwinbee/GLIS.","url_abs":"https://arxiv.org/abs/2407.08931v1","url_pdf":"https://arxiv.org/pdf/2407.08931v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"global-local-collaborative-inference-with-llm","repo_url":"https://github.com/gradiustwinbee/glis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"collaborative-inference","task_name":"Collaborative Inference"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-localization","task_name":"Object Localization"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.08931","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.08931"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/GradiusTwinbee/GLIS","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_honours":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"79bfbbb5dbbbbf3a","entry":"DTCC_loss","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"dtcc_loss.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/dtcc_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79bfbbb5dbbbbf3a"}},{"code_sha256_prefix":"708e6c6c165cb1cd","entry":"adjust_learning_rate","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"engine.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"708e6c6c165cb1cd"}},{"code_sha256_prefix":"da4f8d084fea1b3e","entry":"build_optimizer","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"optimizer.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da4f8d084fea1b3e"}},{"code_sha256_prefix":"c5cf686f87e20400","entry":"cal_sim","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"dtcc_loss.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/dtcc_loss.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5cf686f87e20400"}},{"code_sha256_prefix":"ce7775d94f0bd12d","entry":"compute_learning_rate","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"engine.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce7775d94f0bd12d"}},{"code_sha256_prefix":"f698ca54288e6bee","entry":"dtcc_pc_img_text","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"dtcc_loss.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/dtcc_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f698ca54288e6bee"}},{"code_sha256_prefix":"891b8ebab395921f","entry":"get_clones","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"models/helpers.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/models/helpers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"891b8ebab395921f"}},{"code_sha256_prefix":"90085e8b0f331ce6","entry":"img_norm","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"local_datasets/scannet.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/local_datasets/scannet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90085e8b0f331ce6"}},{"code_sha256_prefix":"69876e2beb68c07e","entry":"load_matrix_from_txt","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"local_datasets/scannet.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/local_datasets/scannet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69876e2beb68c07e"}},{"code_sha256_prefix":"f365b8c11b96a679","entry":"get_color_label","repo":"GradiusTwinbee/GLIS","repo_kind":"official","path":"local_datasets/scannet.py","file_url":"https://github.com/GradiusTwinbee/GLIS/blob/HEAD/local_datasets/scannet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f365b8c11b96a679"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}