{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimath-bridging-visual-and-mathematical","title":"MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models","arxiv_id":"2409.00147","date":"2024-08-30","proceeding":null,"authors":["Shuai Peng","Di Fu","Liangcai Gao","Xiuqin Zhong","Hongguang Fu","Zhi Tang"],"abstract":"The rapid development of large language models (LLMs) has spurred extensive research into their domain-specific capabilities, particularly mathematical reasoning. However, most open-source LLMs focus solely on mathematical reasoning, neglecting the integration with visual injection, despite the fact that many mathematical tasks rely on visual inputs such as geometric diagrams, charts, and function plots. To fill this gap, we introduce \\textbf{MultiMath-7B}, a multimodal large language model that bridges the gap between math and vision. \\textbf{MultiMath-7B} is trained through a four-stage process, focusing on vision-language alignment, visual and math instruction-tuning, and process-supervised reinforcement learning. We also construct a novel, diverse and comprehensive multimodal mathematical dataset, \\textbf{MultiMath-300K}, which spans K-12 levels with image captions and step-wise solutions. MultiMath-7B achieves state-of-the-art (SOTA) performance among open-source models on existing multimodal mathematical benchmarks and also excels on text-only mathematical benchmarks. Our model and dataset are available at {\\textcolor{blue}{\\url{https://github.com/pengshuai-rin/MultiMath}}}.","url_abs":"https://arxiv.org/abs/2409.00147v1","url_pdf":"https://arxiv.org/pdf/2409.00147v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimath-bridging-visual-and-mathematical","repo_url":"https://github.com/pengshuai-rin/multimath","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.00147","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.00147"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pengshuai-rin/multimath","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":8,"ran":2,"ran_violates":3,"unverified":5},"by_repo_kind":{"official":{"samples":18,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ecc1e50a10f04fa0","entry":"create_one_query","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/build_query.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/build_query.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ecc1e50a10f04fa0"}},{"code_sha256_prefix":"6a7cafccc873c34d","entry":"evaluate_code","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/response.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/response.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a7cafccc873c34d"}},{"code_sha256_prefix":"8c9945b50705931d","entry":"extract_boxed_content","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathverse/infer.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathverse/infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c9945b50705931d"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathverse/infer.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathverse/infer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"e18607dae38b30e2","entry":"is_number","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathverse/utils.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathverse/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e18607dae38b30e2"}},{"code_sha256_prefix":"17081a7b73a41850","entry":"load_jsonl","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathverse/utils.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathverse/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17081a7b73a41850"}},{"code_sha256_prefix":"ff5aa85ec44db5ee","entry":"normalize_extracted_answer","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/calculate_score.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/calculate_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff5aa85ec44db5ee"}},{"code_sha256_prefix":"9ffa907555068ff3","entry":"read_json","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/utilities.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/utilities.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ffa907555068ff3"}},{"code_sha256_prefix":"d027e6e473c9fcd4","entry":"refine_caption","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/build_query.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/build_query.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d027e6e473c9fcd4"}},{"code_sha256_prefix":"bfd184123c9012c3","entry":"refine_ocr","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/build_query.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/build_query.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfd184123c9012c3"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathverse/infer.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathverse/infer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"350d0bc5afcc4922","entry":"verify_extraction","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/extract_answer.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/extract_answer.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"350d0bc5afcc4922"}},{"code_sha256_prefix":"d6259b0d9750cac7","entry":"verify_response","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/response.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/response.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d6259b0d9750cac7"}},{"code_sha256_prefix":"a820443033d63826","entry":"extract_boxed_content","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/response.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/response.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a820443033d63826"}},{"code_sha256_prefix":"c28eb86a8d14e4ed","entry":"get_most_similar","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/calculate_score.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/calculate_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c28eb86a8d14e4ed"}},{"code_sha256_prefix":"1694e6d4f46ff13e","entry":"read_csv","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/utilities.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/utilities.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1694e6d4f46ff13e"}},{"code_sha256_prefix":"32ae2fb99638aba7","entry":"read_pandas_csv","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/utilities.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/utilities.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32ae2fb99638aba7"}},{"code_sha256_prefix":"e7a23ea3695f6437","entry":"safe_equal","repo":"pengshuai-rin/multimath","repo_kind":"official","path":"eval_mathvista/calculate_score.py","file_url":"https://github.com/pengshuai-rin/multimath/blob/HEAD/eval_mathvista/calculate_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7a23ea3695f6437"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}