{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omni-math-a-universal-olympiad-level","title":"Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models","arxiv_id":"2410.07985","date":"2024-10-10","proceeding":null,"authors":["Bofei Gao","Feifan Song","Zhe Yang","Zefan Cai","Yibo Miao","Qingxiu Dong","Lei LI","Chenghao Ma","Liang Chen","Runxin Xu","Zhengyang Tang","Benyou Wang","Daoguang Zan","Shanghaoran Quan","Ge Zhang","Lei Sha","Yichang Zhang","Xuancheng Ren","Tianyu Liu","Baobao Chang"],"abstract":"Recent advancements in large language models (LLMs) have led to significant breakthroughs in mathematical reasoning capabilities. However, existing benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g., OpenAI o1 achieves 94.8\\% on MATH dataset), indicating their inadequacy for truly challenging these models. To bridge this gap, we propose a comprehensive and challenging benchmark specifically designed to assess LLMs' mathematical reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks, our dataset focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels, enabling a holistic assessment of model performance in Olympiad-mathematical reasoning. Furthermore, we conducted an in-depth analysis based on this benchmark. Our experimental results show that even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle with highly challenging Olympiad-level problems, with 60.54\\% and 52.55\\% accuracy, highlighting significant challenges in Olympiad-level mathematical reasoning.","url_abs":"https://arxiv.org/abs/2410.07985v3","url_pdf":"https://arxiv.org/pdf/2410.07985v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omni-math-a-universal-olympiad-level","repo_url":"https://github.com/kbsdjames/omni-math","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"omni-math-a-universal-olympiad-level","repo_url":"https://github.com/kbsdjames/omni-math-rule","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.07985","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.07985"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kbsdjames/omni-math","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kbsdjames/omni-math-rule","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_violates":1,"unverified":15},"by_repo_kind":{"official":{"samples":19,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"1525d8bc0fc8d5a0","entry":"generate_completions","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/model_utils.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1525d8bc0fc8d5a0"}},{"code_sha256_prefix":"d338a8cb44aeb6b4","entry":"is_multi_choice","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/math_eval.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/math_eval.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d338a8cb44aeb6b4"}},{"code_sha256_prefix":"b50ea41b74e077ce","entry":"parse_report","repo":"kbsdjames/omni-math","repo_kind":"official","path":"Omni-Judge_eval/get_result.py","file_url":"https://github.com/kbsdjames/omni-math/blob/HEAD/Omni-Judge_eval/get_result.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b50ea41b74e077ce"}},{"code_sha256_prefix":"bd9fc2c45d3b49d6","entry":"update_domain","repo":"kbsdjames/omni-math","repo_kind":"official","path":"GPT_eval/detailed_evaluation/domain_specific_evaluation.py","file_url":"https://github.com/kbsdjames/omni-math/blob/HEAD/GPT_eval/detailed_evaluation/domain_specific_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd9fc2c45d3b49d6"}},{"code_sha256_prefix":"809c0178a7ae90ff","entry":"choice_answer_clean","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/grader.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"809c0178a7ae90ff"}},{"code_sha256_prefix":"81082f28e1440f9d","entry":"compare_numerical_ans","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/math_utils.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81082f28e1440f9d"}},{"code_sha256_prefix":"ed2f4689128f9885","entry":"extract_multi_choice_answer","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/parser.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed2f4689128f9885"}},{"code_sha256_prefix":"3b59d56c92000709","entry":"get_batch_responses","repo":"kbsdjames/omni-math","repo_kind":"official","path":"Omni-Judge_eval/omni_judge.py","file_url":"https://github.com/kbsdjames/omni-math/blob/HEAD/Omni-Judge_eval/omni_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b59d56c92000709"}},{"code_sha256_prefix":"fb37d3c84b18e368","entry":"is_digit","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/grader.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb37d3c84b18e368"}},{"code_sha256_prefix":"581f3949cfdcffa2","entry":"is_execution_success","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/trajectory.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/trajectory.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"581f3949cfdcffa2"}},{"code_sha256_prefix":"1c1060f6e43d5dfa","entry":"is_number","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/math_utils.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c1060f6e43d5dfa"}},{"code_sha256_prefix":"ee71a94d9dd1b8e9","entry":"load_hf_lm_and_tokenizer","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/model_utils.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee71a94d9dd1b8e9"}},{"code_sha256_prefix":"a2095803e48965a1","entry":"main","repo":"kbsdjames/omni-math","repo_kind":"official","path":"Omni-Judge_eval/detailed_evaluation/difficulty_specific_evaluation.py","file_url":"https://github.com/kbsdjames/omni-math/blob/HEAD/Omni-Judge_eval/detailed_evaluation/difficulty_specific_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2095803e48965a1"}},{"code_sha256_prefix":"b7f72b95c867a1ec","entry":"main","repo":"kbsdjames/omni-math","repo_kind":"official","path":"GPT_eval/detailed_evaluation/difficulty_specific_evaluation.py","file_url":"https://github.com/kbsdjames/omni-math/blob/HEAD/GPT_eval/detailed_evaluation/difficulty_specific_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7f72b95c867a1ec"}},{"code_sha256_prefix":"73925940858a62b3","entry":"my_parse_latex","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/math_utils.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73925940858a62b3"}},{"code_sha256_prefix":"7d4b7f8c3875b126","entry":"parse_digits","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/grader.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d4b7f8c3875b126"}},{"code_sha256_prefix":"7e48538d19cc0d8f","entry":"text_to_trajectory","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/trajectory.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/trajectory.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e48538d19cc0d8f"}},{"code_sha256_prefix":"0d7ae684ec6c7537","entry":"trajectory_to_text","repo":"kbsdjames/omni-math-rule","repo_kind":"official","path":"evaluation/trajectory.py","file_url":"https://github.com/kbsdjames/omni-math-rule/blob/HEAD/evaluation/trajectory.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d7ae684ec6c7537"}},{"code_sha256_prefix":"7df9161529866271","entry":"update_domain","repo":"kbsdjames/omni-math","repo_kind":"official","path":"Omni-Judge_eval/detailed_evaluation/domain_specific_evaluation.py","file_url":"https://github.com/kbsdjames/omni-math/blob/HEAD/Omni-Judge_eval/detailed_evaluation/domain_specific_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7df9161529866271"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}