{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understand-think-and-answer-advancing-visual","title":"Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models","arxiv_id":"2505.20753","date":"2025-05-27","proceeding":null,"authors":["Yufei Zhan","Hongyin Zhao","Yousong Zhu","Shurong Zheng","Fan Yang","Ming Tang","Jinqiao Wang"],"abstract":"Large Multimodal Models (LMMs) have recently demonstrated remarkable visual understanding performance on both vision-language and vision-centric tasks. However, they often fall short in integrating advanced, task-specific capabilities for compositional reasoning, which hinders their progress toward truly competent general vision models. To address this, we present a unified visual reasoning mechanism that enables LMMs to solve complicated compositional problems by leveraging their intrinsic capabilities (e.g. grounding and visual understanding capabilities). Different from the previous shortcut learning mechanism, our approach introduces a human-like understanding-thinking-answering process, allowing the model to complete all steps in a single pass forwarding without the need for multiple inferences or external tools. This design bridges the gap between foundational visual capabilities and general question answering, encouraging LMMs to generate faithful and traceable responses for complex visual reasoning. Meanwhile, we curate 334K visual instruction samples covering both general scenes and text-rich scenes and involving multiple foundational visual capabilities. Our trained model, Griffon-R, has the ability of end-to-end automatic understanding, self-thinking, and reasoning answers. Comprehensive experiments show that Griffon-R not only achieves advancing performance on complex visual reasoning benchmarks including VSR and CLEVR, but also enhances multimodal capabilities across various benchmarks like MMBench and ScienceQA. Data, models, and codes will be release at https://github.com/jefferyZhan/Griffon/tree/master/Griffon-R soon.","url_abs":"https://arxiv.org/abs/2505.20753v1","url_pdf":"https://arxiv.org/pdf/2505.20753v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understand-think-and-answer-advancing-visual","repo_url":"https://github.com/jefferyzhan/griffon","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.20753","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.20753"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jefferyzhan/griffon","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":2,"ran_draft_wrong":3,"ran_fixture":1,"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ce749f837424dd8c","entry":"box_iou","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/coor_utils.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/coor_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce749f837424dd8c"}},{"code_sha256_prefix":"592b3c1a88f93d7c","entry":"expand2square","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/mm_utils.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"592b3c1a88f93d7c"}},{"code_sha256_prefix":"dc8fb9b6fb48ac37","entry":"generalized_box_iou","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/coor_utils.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/coor_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc8fb9b6fb48ac37"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/eval/model_vqa.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/eval/model_vqa_qbench.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/eval/model_vqa_qbench.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/mm_utils.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/eval/model_vqa.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"f25030b4707fbdee","entry":"box_xyxy_expand2square","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/coor_utils.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/coor_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f25030b4707fbdee"}},{"code_sha256_prefix":"5ef2d21cca6281db","entry":"collate_fn","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/eval/model_vqa_loader_batch.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/eval/model_vqa_loader_batch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ef2d21cca6281db"}},{"code_sha256_prefix":"29489f5332508b33","entry":"process_images","repo":"jefferyzhan/griffon","repo_kind":"official","path":"griffon/mm_utils.py","file_url":"https://github.com/jefferyzhan/griffon/blob/HEAD/griffon/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29489f5332508b33"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}