{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enhancing-visual-language-modality-alignment","title":"Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement","arxiv_id":"2405.15973","date":"2024-05-24","proceeding":null,"authors":["Xiyao Wang","Jiuhai Chen","Zhaoyang Wang","YuHang Zhou","Yiyang Zhou","Huaxiu Yao","Tianyi Zhou","Tom Goldstein","Parminder Bhatia","Furong Huang","Cao Xiao"],"abstract":"Large vision-language models (LVLMs) have achieved impressive results in various visual question-answering and reasoning tasks through vision instruction tuning on specific datasets. However, there is still significant room for improvement in the alignment between visual and language modalities. Previous methods to enhance this alignment typically require external models or data, heavily depending on their capabilities and quality, which inevitably sets an upper bound on performance. In this paper, we propose SIMA, a framework that enhances visual and language modality alignment through self-improvement, eliminating the needs for external models or data. SIMA leverages prompts from existing vision instruction tuning datasets to self-generate responses and employs an in-context self-critic mechanism to select response pairs for preference tuning. The key innovation is the introduction of three vision metrics during the in-context self-critic process, which can guide the LVLM in selecting responses that enhance image comprehension. Through experiments across 14 hallucination and comprehensive benchmarks, we demonstrate that SIMA not only improves model performance across all benchmarks but also achieves superior modality alignment, outperforming previous approaches.","url_abs":"https://arxiv.org/abs/2405.15973v3","url_pdf":"https://arxiv.org/pdf/2405.15973v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"enhancing-visual-language-modality-alignment","repo_url":"https://github.com/umd-huang-lab/sima","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"enhancing-visual-language-modality-alignment","repo_url":"https://github.com/yiyangzhou/csr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"image-comprehension","task_name":"Image Comprehension"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"LLaVA-1.5-7B (+ SIMA)","rank_in_archive_order":193,"of":231,"metrics":{"GPT-4 score":"31.6","Params":"7B"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.15973","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15973"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yiyangzhou/csr","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/umd-huang-lab/sima","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran_violates":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"umd-huang-lab/sima","repo_kind":"official","path":"llava/eval/model_vqa_loader.py","file_url":"https://github.com/umd-huang-lab/sima/blob/HEAD/llava/eval/model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"umd-huang-lab/sima","repo_kind":"official","path":"llava/sima/model_self_rewarding.py","file_url":"https://github.com/umd-huang-lab/sima/blob/HEAD/llava/sima/model_self_rewarding.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"umd-huang-lab/sima","repo_kind":"official","path":"llava/eval/model_vqa_mmbench.py","file_url":"https://github.com/umd-huang-lab/sima/blob/HEAD/llava/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"umd-huang-lab/sima","repo_kind":"official","path":"predict.py","file_url":"https://github.com/umd-huang-lab/sima/blob/HEAD/predict.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"umd-huang-lab/sima","repo_kind":"official","path":"llava/sima/model_self_rewarding.py","file_url":"https://github.com/umd-huang-lab/sima/blob/HEAD/llava/sima/model_self_rewarding.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}