{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/logicvista-multimodal-llm-logical-reasoning","title":"LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts","arxiv_id":"2407.04973","date":"2024-07-06","proceeding":null,"authors":["Yijia Xiao","Edward Sun","Tianyu Liu","Wei Wang"],"abstract":"We propose LogicVista, an evaluation benchmark that assesses the integrated logical reasoning capabilities of multimodal large language models (MLLMs) in Visual contexts. Recent advancements in MLLMs have demonstrated various fascinating abilities, from crafting poetry based on an image to performing mathematical reasoning. However, there is still a lack of systematic evaluation of MLLMs' proficiency in logical reasoning tasks, which are essential for activities like navigation and puzzle-solving. Thus we evaluate general logical cognition abilities across 5 logical reasoning tasks encompassing 9 different capabilities, using a sample of 448 multiple-choice questions. Each question is annotated with the correct answer and the human-written reasoning behind the selection, enabling both open-ended and multiple-choice evaluation. A total of 8 MLLMs are comprehensively evaluated using LogicVista. Code and Data Available at https://github.com/Yijia-Xiao/LogicVista.","url_abs":"https://arxiv.org/abs/2407.04973v1","url_pdf":"https://arxiv.org/pdf/2407.04973v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"logicvista-multimodal-llm-logical-reasoning","repo_url":"https://github.com/yijia-xiao/logicvista","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.04973","atlas_url":"https://app.syntology.ai/?focus=2407.04973","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.04973"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yijia-xiao/logicvista","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0ddad62c4329b3ef","entry":"get_fields","repo":"yijia-xiao/logicvista","repo_kind":"official","path":"label_studio/core/bulk_update_utils.py","file_url":"https://github.com/yijia-xiao/logicvista/blob/HEAD/label_studio/core/bulk_update_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ddad62c4329b3ef"}},{"code_sha256_prefix":"f942b532e3a9a0fa","entry":"override_report_only_csp","repo":"yijia-xiao/logicvista","repo_kind":"official","path":"label_studio/core/decorators.py","file_url":"https://github.com/yijia-xiao/logicvista/blob/HEAD/label_studio/core/decorators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f942b532e3a9a0fa"}},{"code_sha256_prefix":"e74c059f314bd944","entry":"flatten","repo":"yijia-xiao/logicvista","repo_kind":"official","path":"label_studio/core/bulk_update_utils.py","file_url":"https://github.com/yijia-xiao/logicvista/blob/HEAD/label_studio/core/bulk_update_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e74c059f314bd944"}},{"code_sha256_prefix":"3c85838d61a47ec1","entry":"parse_input_args","repo":"yijia-xiao/logicvista","repo_kind":"official","path":"label_studio/core/argparser.py","file_url":"https://github.com/yijia-xiao/logicvista/blob/HEAD/label_studio/core/argparser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c85838d61a47ec1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}