{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llava-grounding-grounded-visual-chat-with","title":"LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models","arxiv_id":"2312.02949","date":"2023-12-05","proceeding":null,"authors":["Hao Zhang","Hongyang Li","Feng Li","Tianhe Ren","Xueyan Zou","Shilong Liu","Shijia Huang","Jianfeng Gao","Lei Zhang","Chunyuan Li","Jianwei Yang"],"abstract":"With the recent significant advancements in large multi-modal models (LMMs), the importance of their grounding capability in visual chat is increasingly recognized. Despite recent efforts to enable LMMs to support grounding, their capabilities for grounding and chat are usually separate, and their chat performance drops dramatically when asked to ground. The problem is the lack of a dataset for grounded visual chat (GVC). Existing grounding datasets only contain short captions. To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities. To better evaluate the GVC capabilities, we have introduced a benchmark called Grounding-Bench. Additionally, we have proposed a model design that can support GVC and various types of visual prompts by connecting segmentation models with language models. Experimental results demonstrate that our model outperforms other LMMs on Grounding-Bench. Furthermore, our model achieves competitive performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K Entities. Our code will be released at https://github.com/UX-Decoder/LLaVA-Grounding .","url_abs":"https://arxiv.org/abs/2312.02949v1","url_pdf":"https://arxiv.org/pdf/2312.02949v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llava-grounding-grounded-visual-chat-with","repo_url":"https://github.com/ux-decoder/llava-grounding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02949","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02949"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ux-decoder/llava-grounding","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9acc7c1533708433","entry":"filter_empty_box_mask","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"gradio_demo/LLaVA_G_Demo.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/gradio_demo/LLaVA_G_Demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9acc7c1533708433"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"8aaa8f19ee3c5bf9","entry":"get_image_name","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"gradio_demo/LLaVA_G_Demo.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/gradio_demo/LLaVA_G_Demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8aaa8f19ee3c5bf9"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"1df990c375318896","entry":"process_images","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1df990c375318896"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"279c6e1d7ca04f75","entry":"get_peft_state_maybe_zero_3","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"llava/train/train_grounding_1st.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/llava/train/train_grounding_1st.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"279c6e1d7ca04f75"}},{"code_sha256_prefix":"1c53657305b66e9f","entry":"get_peft_state_non_lora_maybe_zero_3","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"llava/train/train_grounding_1st.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/llava/train/train_grounding_1st.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c53657305b66e9f"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"ux-decoder/llava-grounding","repo_kind":"official","path":"llava/train/train_grounding_1st.py","file_url":"https://github.com/ux-decoder/llava-grounding/blob/HEAD/llava/train/train_grounding_1st.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}