{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lego-language-enhanced-multi-modal-grounding","title":"GroundingGPT:Language Enhanced Multi-modal Grounding Model","arxiv_id":"2401.06071","date":"2024-01-11","proceeding":null,"authors":["Zhaowei Li","Qi Xu","Dong Zhang","Hang Song","Yiqing Cai","Qi Qi","Ran Zhou","Junting Pan","Zefeng Li","Van Tu Vu","Zhida Huang","Tao Wang"],"abstract":"Multi-modal large language models have demonstrated impressive performance across various tasks in different modalities. However, existing multi-modal models primarily emphasize capturing global information within each modality while neglecting the importance of perceiving local information across modalities. Consequently, these models lack the ability to effectively understand the fine-grained details of input data, limiting their performance in tasks that require a more nuanced understanding. To address this limitation, there is a compelling need to develop models that enable fine-grained understanding across multiple modalities, thereby enhancing their applicability to a wide range of tasks. In this paper, we propose GroundingGPT, a language enhanced multi-modal grounding model. Beyond capturing global information like other multi-modal models, our proposed model excels at tasks demanding a detailed understanding of local information within the input. It demonstrates precise identification and localization of specific regions in images or moments in videos. To achieve this objective, we design a diversified dataset construction pipeline, resulting in a multi-modal, multi-granularity dataset for model training. The code, dataset, and demo of our model can be found at https: //github.com/lzw-lzw/GroundingGPT.","url_abs":"https://arxiv.org/abs/2401.06071v5","url_pdf":"https://arxiv.org/pdf/2401.06071v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lego-language-enhanced-multi-modal-grounding","repo_url":"https://github.com/lzw-lzw/groundinggpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"lego-language-enhanced-multi-modal-grounding","repo_url":"https://github.com/lzw-lzw/lego","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.06071","atlas_url":"https://app.syntology.ai/?focus=2401.06071","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.06071"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lzw-lzw/lego","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lzw-lzw/groundinggpt","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"video_llama/models/blip2.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/video_llama/models/blip2.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"lego/mm_utils.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/lego/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"f04e8e006a8831c9","entry":"load_image_square","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"lego/mm_utils.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/lego/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f04e8e006a8831c9"}},{"code_sha256_prefix":"b8030ce2d52399ba","entry":"postprocess_box","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"lego/mm_utils.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/lego/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8030ce2d52399ba"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"video_llama/models/base_model.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/video_llama/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"video_llama/models/base_model.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/video_llama/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"lego/utils.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/lego/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"e2e5c3a95a6aebb4","entry":"save_image_to_local","repo":"lzw-lzw/lego","repo_kind":"official","path":"lego/serve/gradio_web_server.py","file_url":"https://github.com/lzw-lzw/lego/blob/HEAD/lego/serve/gradio_web_server.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2e5c3a95a6aebb4"}},{"code_sha256_prefix":"e75fa36a185a618d","entry":"save_sound_to_local","repo":"lzw-lzw/lego","repo_kind":"official","path":"lego/serve/gradio_web_server.py","file_url":"https://github.com/lzw-lzw/lego/blob/HEAD/lego/serve/gradio_web_server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e75fa36a185a618d"}},{"code_sha256_prefix":"22930d58d32e2d08","entry":"save_video_to_local","repo":"lzw-lzw/lego","repo_kind":"official","path":"lego/serve/gradio_web_server.py","file_url":"https://github.com/lzw-lzw/lego/blob/HEAD/lego/serve/gradio_web_server.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22930d58d32e2d08"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"lzw-lzw/groundinggpt","repo_kind":"official","path":"lego/utils.py","file_url":"https://github.com/lzw-lzw/groundinggpt/blob/HEAD/lego/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}