{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uni-mumer-unified-multi-task-fine-tuning-of","title":"Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition","arxiv_id":"2505.23566","date":"2025-05-29","proceeding":null,"authors":["Yu Li","Jin Jiang","Jianhua Zhu","Shuai Peng","Baole Wei","Yuxuan Zhou","Liangcai Gao"],"abstract":"Handwritten Mathematical Expression Recognition (HMER) remains a persistent challenge in Optical Character Recognition (OCR) due to the inherent freedom of symbol layout and variability in handwriting styles. Prior methods have faced performance bottlenecks, proposing isolated architectural modifications that are difficult to integrate coherently into a unified framework. Meanwhile, recent advances in pretrained vision-language models (VLMs) have demonstrated strong cross-task generalization, offering a promising foundation for developing unified solutions. In this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM for the HMER task without modifying its architecture, effectively injecting domain-specific knowledge into a generalist framework. Our method integrates three data-driven tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structured spatial reasoning, Error-Driven Learning (EDL) for reducing confusion among visually similar characters, and Symbol Counting (SC) for improving recognition consistency in long expressions. Experiments on the CROHME and HME100K datasets show that Uni-MuMER achieves new state-of-the-art performance, surpassing the best lightweight specialized model SSAN by 16.31% and the top-performing VLM Gemini2.5-flash by 24.42% in the zero-shot setting. Our datasets, models, and code are open-sourced at: https://github.com/BFlameSwift/Uni-MuMER","url_abs":"https://arxiv.org/abs/2505.23566v1","url_pdf":"https://arxiv.org/pdf/2505.23566v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uni-mumer-unified-multi-task-fine-tuning-of","repo_url":"https://github.com/bflameswift/uni-mumer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"handwritten-mathmatical-expression","task_name":"Handwritten Mathmatical Expression Recognition"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/handwritten-mathmatical-expression","task":"Handwritten Mathmatical Expression Recognition","dataset":"CROHME 2014","model":"Uni-MuMER","rank_in_archive_order":1,"of":17,"metrics":{"ExpRate":"82.05"},"uses_additional_data":true},{"leaderboard":"/sota/handwritten-mathmatical-expression","task":"Handwritten Mathmatical Expression Recognition","dataset":"CROHME 2014","model":"Uni-MuMER","rank_in_archive_order":2,"of":17,"metrics":{"ExpRate":"75.36"},"uses_additional_data":false},{"leaderboard":"/sota/handwritten-mathmatical-expression-1","task":"Handwritten Mathmatical Expression Recognition","dataset":"CROHME 2016","model":"Uni-MuMER","rank_in_archive_order":1,"of":16,"metrics":{"ExpRate":"77.94"},"uses_additional_data":true},{"leaderboard":"/sota/handwritten-mathmatical-expression-2","task":"Handwritten Mathmatical Expression Recognition","dataset":"CROHME 2019","model":"Uni-MuMER","rank_in_archive_order":1,"of":14,"metrics":{"ExpRate":"79.23"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.23566","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.23566"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/BFlameSwift/Uni-MuMER","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bflameswift/uni-mumer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_honours":2,"ran_fixture":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"902d6fc90dfc3c3b","entry":"Node","repo":"bflameswift/uni-mumer","repo_kind":"official","path":"preprocess/unimumer_tree/latex2tree.py","file_url":"https://github.com/bflameswift/uni-mumer/blob/HEAD/preprocess/unimumer_tree/latex2tree.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"902d6fc90dfc3c3b"}},{"code_sha256_prefix":"51b08bef9ae97ab4","entry":"findendenv","repo":"bflameswift/uni-mumer","repo_kind":"official","path":"preprocess/unimumer_tree/latex2tree.py","file_url":"https://github.com/bflameswift/uni-mumer/blob/HEAD/preprocess/unimumer_tree/latex2tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51b08bef9ae97ab4"}},{"code_sha256_prefix":"fb7b689b9588f582","entry":"findnextbracket","repo":"bflameswift/uni-mumer","repo_kind":"official","path":"preprocess/unimumer_tree/latex2tree.py","file_url":"https://github.com/bflameswift/uni-mumer/blob/HEAD/preprocess/unimumer_tree/latex2tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb7b689b9588f582"}},{"code_sha256_prefix":"75ff77728a39652d","entry":"get_symbol_counts","repo":"bflameswift/uni-mumer","repo_kind":"official","path":"preprocess/make_unimumer_tree_data.py","file_url":"https://github.com/bflameswift/uni-mumer/blob/HEAD/preprocess/make_unimumer_tree_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75ff77728a39652d"}},{"code_sha256_prefix":"b8f2bff4488c69d5","entry":"latex2Tree","repo":"bflameswift/uni-mumer","repo_kind":"official","path":"preprocess/unimumer_tree/latex2tree.py","file_url":"https://github.com/bflameswift/uni-mumer/blob/HEAD/preprocess/unimumer_tree/latex2tree.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8f2bff4488c69d5"}},{"code_sha256_prefix":"92e5c6e686c8d6c2","entry":"make_count_dict2str","repo":"bflameswift/uni-mumer","repo_kind":"official","path":"preprocess/make_unimumer_tree_data.py","file_url":"https://github.com/bflameswift/uni-mumer/blob/HEAD/preprocess/make_unimumer_tree_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92e5c6e686c8d6c2"}},{"code_sha256_prefix":"23ca323607bb2020","entry":"validate_record","repo":"bflameswift/uni-mumer","repo_kind":"official","path":"preprocess/make_unimumer_tree_data.py","file_url":"https://github.com/bflameswift/uni-mumer/blob/HEAD/preprocess/make_unimumer_tree_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23ca323607bb2020"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}