{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unibench-visual-reasoning-requires-rethinking","title":"UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling","arxiv_id":"2408.04810","date":"2024-08-09","proceeding":null,"authors":["Haider Al-Tahan","Quentin Garrido","Randall Balestriero","Diane Bouchacourt","Caner Hazirbas","Mark Ibrahim"],"abstract":"Significant research efforts have been made to scale and improve vision-language model (VLM) training approaches. Yet, with an ever-growing number of benchmarks, researchers are tasked with the heavy burden of implementing each protocol, bearing a non-trivial computational cost, and making sense of how all these benchmarks translate into meaningful axes of progress. To facilitate a systematic evaluation of VLM progress, we introduce UniBench: a unified implementation of 50+ VLM benchmarks spanning a comprehensive range of carefully categorized capabilities from object recognition to spatial awareness, counting, and much more. We showcase the utility of UniBench for measuring progress by evaluating nearly 60 publicly available vision-language models, trained on scales of up to 12.8B samples. We find that while scaling training data or model size can boost many vision-language model capabilities, scaling offers little benefit for reasoning or relations. Surprisingly, we also discover today's best VLMs struggle on simple digit recognition and counting tasks, e.g. MNIST, which much simpler networks can solve. Where scale falls short, we find that more precise interventions, such as data quality or tailored-learning objectives offer more promise. For practitioners, we also offer guidance on selecting a suitable VLM for a given application. Finally, we release an easy-to-run UniBench code-base with the full set of 50+ benchmarks and comparisons across 59 models as well as a distilled, representative set of benchmarks that runs in 5 minutes on a single GPU.","url_abs":"https://arxiv.org/abs/2408.04810v1","url_pdf":"https://arxiv.org/pdf/2408.04810v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unibench-visual-reasoning-requires-rethinking","repo_url":"https://github.com/facebookresearch/unibench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.04810","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.04810"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/unibench","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":5,"unverified":4},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"acb03ffc90c5d69e","entry":"df_to_table","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/common_utils/utils.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/common_utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"acb03ffc90c5d69e"}},{"code_sha256_prefix":"157cbc5ff7648503","entry":"get_benchmark_info","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/benchmarks_zoo/registry.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/benchmarks_zoo/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"157cbc5ff7648503"}},{"code_sha256_prefix":"ce4164ce34ef669f","entry":"get_model_info","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/models_zoo/registry.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/models_zoo/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ce4164ce34ef669f"}},{"code_sha256_prefix":"46d22fdff321711d","entry":"register_benchmark","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/benchmarks_zoo/registry.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/benchmarks_zoo/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"46d22fdff321711d"}},{"code_sha256_prefix":"443a61be3b02f5a2","entry":"register_model","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/models_zoo/registry.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/models_zoo/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"443a61be3b02f5a2"}},{"code_sha256_prefix":"44c30f2c1af83047","entry":"get_benchmark_mappings","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/common_utils/utils.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/common_utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"44c30f2c1af83047"}},{"code_sha256_prefix":"c2855847e2b846d7","entry":"get_model_mappings","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/common_utils/utils.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/common_utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c2855847e2b846d7"}},{"code_sha256_prefix":"d303d39673c61784","entry":"load_benchmark","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/benchmarks_zoo/registry.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/benchmarks_zoo/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d303d39673c61784"}},{"code_sha256_prefix":"b81a08983be5a4ee","entry":"load_model","repo":"facebookresearch/unibench","repo_kind":"official","path":"unibench/models_zoo/registry.py","file_url":"https://github.com/facebookresearch/unibench/blob/HEAD/unibench/models_zoo/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b81a08983be5a4ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}