{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-ended-vqa-benchmarking-of-vision","title":"Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy","arxiv_id":"2402.07270","date":"2024-02-11","proceeding":null,"authors":["Simon Ging","María A. Bravo","Thomas Brox"],"abstract":"The evaluation of text-generative vision-language models is a challenging yet crucial endeavor. By addressing the limitations of existing Visual Question Answering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models' capabilities. We propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models. To improve the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category. Finally, we compare traditional NLP and LLM-based metrics for the problem of evaluating model predictions given ground-truth answers. We perform a human evaluation study upon which we base our decision on the final metric. We apply our benchmark to a suite of vision-language models and show a detailed comparison of their abilities on object, action, and attribute classification. Our contributions aim to lay the foundation for more precise and meaningful assessments, facilitating targeted progress in the exciting field of vision-language modeling.","url_abs":"https://arxiv.org/abs/2402.07270v2","url_pdf":"https://arxiv.org/pdf/2402.07270v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-ended-vqa-benchmarking-of-vision","repo_url":"https://github.com/lmb-freiburg/ovqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"open-vocabulary-attribute-detection","task_name":"Open Vocabulary Attribute Detection"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-vqa-on-activitynet-1","task":"Visual Question Answering (VQA)","dataset":"ActivityNet","model":"BLIP-2 T5","rank_in_archive_order":1,"of":1,"metrics":{"ClipMatch@1":"53.39","ClipMatch@5":"74.71","Contains":"15.70","ExactMatch":"7.07","Follow-up ClipMatch@1":"62.02","Follow-up ClipMatch@5":"75.13","Follow-up Contains":"18.09","Follow-up ExactMatch":"8.84"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on-coco","task":"Visual Question Answering (VQA)","dataset":"COCO","model":"InstructBLIP Vicuna","rank_in_archive_order":1,"of":1,"metrics":{"ClipMatch@1":"59.58","ClipMatch@5":"73.32","Contains":"27.52","ExactMatch":"26.50"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on-imagenet","task":"Visual Question Answering (VQA)","dataset":"ImageNet","model":"BLIP-2 OPT","rank_in_archive_order":1,"of":1,"metrics":{"ClipMatch@1":"57.10","ClipMatch@5":"77.24","Contains":"35.49","ExactMatch":"0.87","Follow-up ClipMatch@1":"67.22","Follow-up ClipMatch@5":"83.54","Follow-up Contains":"40.31","Follow-up ExactMatch":"2.54"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-vqa-on-ovad","task":"Visual Question Answering (VQA)","dataset":"OVAD benchmark","model":"BLIP","rank_in_archive_order":1,"of":1,"metrics":{"Contains w. Synonyms":"45.70","ExactMatch w. Synonyms":"36.99"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.07270","atlas_url":"https://app.syntology.ai/?focus=2402.07270","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.07270"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lmb-freiburg/ovqa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3cd4a50601da36c4","entry":"apply_layer_decay","repo":"lmb-freiburg/ovqa","repo_kind":"official","path":"ovqa/optim_layer_decay.py","file_url":"https://github.com/lmb-freiburg/ovqa/blob/HEAD/ovqa/optim_layer_decay.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cd4a50601da36c4"}},{"code_sha256_prefix":"e1cf91cb1d73f4b7","entry":"create_prompt","repo":"lmb-freiburg/ovqa","repo_kind":"official","path":"ovqa/create_prompt.py","file_url":"https://github.com/lmb-freiburg/ovqa/blob/HEAD/ovqa/create_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1cf91cb1d73f4b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}