{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-16256","title":"Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap","arxiv_id":"2604.16256","date":"2026-04-17","proceeding":null,"authors":["Yige Xu","Yongjie Wang","Zizhuo Wu","Kaisong Song","Jun Lin","Zhiqi Shen"],"abstract":"Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks. However, it remains unclear whether the superior performance of VLMs stems from genuine vision-grounded reasoning or relies predominantly on the reasoning capabilities of their textual backbones. To systematically measure this, we introduce CrossMath, a novel multimodal reasoning benchmark designed for controlled cross-modal comparisons. Specifically, we construct each problem in text-only, image-only, and image+text formats guaranteeing identical task-relevant information, verified by human annotators. This rigorous alignment effectively isolates modality-specific reasoning differences while eliminating confounding factors such as information mismatch. Extensive evaluation of state-of-the-art VLMs reveals a consistent phenomenon: a substantial performance gap between textual and visual reasoning. Notably, VLMs excel with text-only inputs, whereas incorporating visual data (image+text) frequently degrades performance compared to the text-only baseline. These findings indicate that current VLMs conduct reasoning primarily in the textual space, with limited genuine reliance on visual evidence. To mitigate this limitation, we curate a CrossMath training set for VLM fine-tuning. Empirical evaluations demonstrate that fine-tuning on this training set significantly boosts reasoning performance across all individual and joint modalities, while yielding robust gains on two general visual reasoning tasks. Source code is available at https://github.com/xuyige/CrossMath.","url_abs":"https://arxiv.org/abs/2604.16256","url_pdf":"https://arxiv.org/pdf/2604.16256","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.16256","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.16256"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/xuyige/CrossMath","reach":null}],"summary":{"unverified":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"c53d52049bcc871f","entry":"extract_answer","repo":"xuyige/CrossMath","repo_kind":"found_in_text","path":"batch_inference_qwen35.py","file_url":"https://github.com/xuyige/CrossMath/blob/HEAD/batch_inference_qwen35.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c53d52049bcc871f"}},{"code_sha256_prefix":"7d1c8544d5e2cf05","entry":"extract_answer_tokens","repo":"xuyige/CrossMath","repo_kind":"found_in_text","path":"batch_inference_qwen35.py","file_url":"https://github.com/xuyige/CrossMath/blob/HEAD/batch_inference_qwen35.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7d1c8544d5e2cf05"}},{"code_sha256_prefix":"d33376cf99af8e91","entry":"normalize_pred_line","repo":"xuyige/CrossMath","repo_kind":"found_in_text","path":"calc_metric.py","file_url":"https://github.com/xuyige/CrossMath/blob/HEAD/calc_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d33376cf99af8e91"}},{"code_sha256_prefix":"b17c5dd5cbb2e714","entry":"normalize_step_category","repo":"xuyige/CrossMath","repo_kind":"found_in_text","path":"calc_metric.py","file_url":"https://github.com/xuyige/CrossMath/blob/HEAD/calc_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b17c5dd5cbb2e714"}},{"code_sha256_prefix":"26b429fe4c4d9663","entry":"parse_pipe_separated","repo":"xuyige/CrossMath","repo_kind":"found_in_text","path":"calc_metric.py","file_url":"https://github.com/xuyige/CrossMath/blob/HEAD/calc_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"26b429fe4c4d9663"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}