{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unveiling-the-tapestry-of-consistency-in","title":"Unveiling the Tapestry of Consistency in Large Vision-Language Models","arxiv_id":"2405.14156","date":"2024-05-23","proceeding":null,"authors":["Yuan Zhang","Fei Xiao","Tao Huang","Chun-Kai Fan","Hongyuan Dong","Jiawen Li","Jiacong Wang","Kuan Cheng","Shanghang Zhang","Haoyuan Guo"],"abstract":"Large vision-language models (LVLMs) have recently achieved rapid progress, exhibiting great perception and reasoning abilities concerning visual information. However, when faced with prompts in different sizes of solution spaces, LVLMs fail to always give consistent answers regarding the same knowledge point. This inconsistency of answers between different solution spaces is prevalent in LVLMs and erodes trust. To this end, we provide a multi-modal benchmark ConBench, to intuitively analyze how LVLMs perform when the solution space of a prompt revolves around a knowledge point. Based on the ConBench tool, we are the first to reveal the tapestry and get the following findings: (1) In the discriminate realm, the larger the solution space of the prompt, the lower the accuracy of the answers. (2) Establish the relationship between the discriminative and generative realms: the accuracy of the discriminative question type exhibits a strong positive correlation with its Consistency with the caption. (3) Compared to open-source models, closed-source models exhibit a pronounced bias advantage in terms of Consistency. Eventually, we ameliorate the consistency of LVLMs by trigger-based diagnostic refinement, indirectly improving the performance of their caption. We hope this paper will accelerate the research community in better evaluating their models and encourage future advancements in the consistency domain. The project is available at https://github.com/foundation-multimodal-models/ConBench.","url_abs":"https://arxiv.org/abs/2405.14156v4","url_pdf":"https://arxiv.org/pdf/2405.14156v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unveiling-the-tapestry-of-consistency-in","repo_url":"https://github.com/foundation-multimodal-models/conbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.14156","atlas_url":"https://app.syntology.ai/?focus=2405.14156","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14156"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/foundation-multimodal-models/conbench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cdf4256741c2d7e3","entry":"GPTAnswer","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"Score.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/Score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdf4256741c2d7e3"}},{"code_sha256_prefix":"57e95166ade5f25c","entry":"build_transform","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"eval/InternVL-Chat-V1-5-26B.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/eval/InternVL-Chat-V1-5-26B.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57e95166ade5f25c"}},{"code_sha256_prefix":"d610b5eabe0c9db1","entry":"dynamic_preprocess","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"eval/InternVL-Chat-V1-5-26B.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/eval/InternVL-Chat-V1-5-26B.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d610b5eabe0c9db1"}},{"code_sha256_prefix":"f41cb1a19b154297","entry":"encode_image","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"eval/GPT-4o.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/eval/GPT-4o.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f41cb1a19b154297"}},{"code_sha256_prefix":"bd77f5f8067f18e9","entry":"find_closest_aspect_ratio","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"eval/InternVL-Chat-V1-5-26B.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/eval/InternVL-Chat-V1-5-26B.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd77f5f8067f18e9"}},{"code_sha256_prefix":"59b41eb82bbd9fae","entry":"testOpenaiChatCompletions","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"Score.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/Score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59b41eb82bbd9fae"}},{"code_sha256_prefix":"34b2c537fc90ba96","entry":"testOpenaiChatCompletions","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"eval/GPT_4V.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/eval/GPT_4V.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34b2c537fc90ba96"}},{"code_sha256_prefix":"6364f5bae014de87","entry":"testOpenaiChatCompletions","repo":"foundation-multimodal-models/conbench","repo_kind":"official","path":"eval/GPT-4o.py","file_url":"https://github.com/foundation-multimodal-models/conbench/blob/HEAD/eval/GPT-4o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6364f5bae014de87"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}