{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/synthesize-diagnose-and-optimize-towards-fine","title":"Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding","arxiv_id":"2312.00081","date":"2023-11-30","proceeding":null,"authors":["Wujian Peng","Sicheng Xie","Zuyao You","Shiyi Lan","Zuxuan Wu"],"abstract":"Vision language models (VLM) have demonstrated remarkable performance across various downstream tasks. However, understanding fine-grained visual-linguistic concepts, such as attributes and inter-object relationships, remains a significant challenge. While several benchmarks aim to evaluate VLMs in finer granularity, their primary focus remains on the linguistic aspect, neglecting the visual dimension. Here, we highlight the importance of evaluating VLMs from both a textual and visual perspective. We introduce a progressive pipeline to synthesize images that vary in a specific attribute while ensuring consistency in all other aspects. Utilizing this data engine, we carefully design a benchmark, SPEC, to diagnose the comprehension of object size, position, existence, and count. Subsequently, we conduct a thorough evaluation of four leading VLMs on SPEC. Surprisingly, their performance is close to random guess, revealing significant limitations. With this in mind, we propose a simple yet effective approach to optimize VLMs in fine-grained understanding, achieving significant improvements on SPEC without compromising the zero-shot performance. Results on two additional fine-grained benchmarks also show consistent improvements, further validating the transferability of our approach. Code and data are available at https://github.com/wjpoom/SPEC.","url_abs":"https://arxiv.org/abs/2312.00081v2","url_pdf":"https://arxiv.org/pdf/2312.00081v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"synthesize-diagnose-and-optimize-towards-fine","repo_url":"https://github.com/wjpoom/spec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"compositional-zero-shot-learning","task_name":"Compositional Zero-Shot Learning"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.00081","atlas_url":"https://app.syntology.ai/?focus=2312.00081","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.00081"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wjpoom/spec","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"ran":1,"unverified":7},"by_repo_kind":{"official":{"samples":11,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"aa81d03f07735ab8","entry":"compute_acc","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/utils.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa81d03f07735ab8"}},{"code_sha256_prefix":"ad12a494674d23fb","entry":"compute_n_params","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/utils.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad12a494674d23fb"}},{"code_sha256_prefix":"0097e95cd9091012","entry":"get_data","repo":"wjpoom/spec","repo_kind":"official","path":"spec/dataset.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0097e95cd9091012"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/vit.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/blip_retrieval.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/blip_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"1bd954c1ba107cb9","entry":"blip_decoder","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/blip.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/blip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1bd954c1ba107cb9"}},{"code_sha256_prefix":"16c013eb492b2d8c","entry":"blip_feature_extractor","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/blip.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/blip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16c013eb492b2d8c"}},{"code_sha256_prefix":"84d0b6c25badc1c3","entry":"blip_itm","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/blip_itm.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/blip_itm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84d0b6c25badc1c3"}},{"code_sha256_prefix":"099da43f6db7e41d","entry":"blip_retrieval","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/blip_retrieval.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/blip_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"099da43f6db7e41d"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/blip_pretrain.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/blip_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"f837c334ddfa8af4","entry":"create_vit","repo":"wjpoom/spec","repo_kind":"official","path":"spec/models/blip_utils/blip.py","file_url":"https://github.com/wjpoom/spec/blob/HEAD/spec/models/blip_utils/blip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f837c334ddfa8af4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}