{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/leveraging-online-olympiad-level-math","title":"Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation","arxiv_id":"2501.14275","date":"2025-01-24","proceeding":null,"authors":["Sadegh Mahdavi","Muchen Li","Kaiwen Liu","Christos Thrampoulidis","Leonid Sigal","Renjie Liao"],"abstract":"Advances in Large Language Models (LLMs) have sparked interest in their ability to solve Olympiad-level math problems. However, the training and evaluation of these models are constrained by the limited size and quality of available datasets, as creating large-scale data for such advanced problems requires extensive effort from human experts. In addition, current benchmarks are prone to contamination, leading to unreliable evaluations. In this paper, we present an automated pipeline that leverages the rich resources of the Art of Problem Solving (AoPS) forum, which predominantly features Olympiad-level problems and community-driven solutions. Using open-source LLMs, we develop a method to extract question-answer pairs from the forum, resulting in AoPS-Instruct, a dataset of more than 600,000 high-quality QA pairs. Our experiments demonstrate that fine-tuning LLMs on AoPS-Instruct improves their reasoning abilities across various benchmarks. Moreover, we build an automatic pipeline that introduces LiveAoPSBench, an evolving evaluation set with timestamps, derived from the latest forum data, providing a contamination-resistant benchmark for assessing LLM performance. Notably, we observe a significant decline in LLM performance over time, suggesting their success on older examples may stem from pre-training exposure rather than true reasoning ability. Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning, offering valuable insights into the capabilities and limitations of LLMs in this domain. Our benchmark and code is available at https://github.com/DSL-Lab/aops","url_abs":"https://arxiv.org/abs/2501.14275v1","url_pdf":"https://arxiv.org/pdf/2501.14275v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leveraging-online-olympiad-level-math","repo_url":"https://github.com/dsl-lab/aops","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"leveraging-online-olympiad-level-math","repo_url":"https://github.com/multimodal-art-projection/criticlean","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.14275","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.14275"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dsl-lab/aops","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/multimodal-art-projection/criticlean","reach":{"status":"ok"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"403712b231130602","entry":"build_prompt","repo":"dsl-lab/aops","repo_kind":"official","path":"classify_aops.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/classify_aops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"403712b231130602"}},{"code_sha256_prefix":"fe1c5e8181b64c3c","entry":"find_post_year","repo":"dsl-lab/aops","repo_kind":"official","path":"aops_crawler/clean_raw.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/aops_crawler/clean_raw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe1c5e8181b64c3c"}},{"code_sha256_prefix":"74bdf493edad4b7d","entry":"get_instruct_formalize_answer","repo":"dsl-lab/aops","repo_kind":"official","path":"aops/few_shot_prompt.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/aops/few_shot_prompt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74bdf493edad4b7d"}},{"code_sha256_prefix":"a0376cee46c008fc","entry":"get_instruct_formalize_question","repo":"dsl-lab/aops","repo_kind":"official","path":"aops/few_shot_prompt.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/aops/few_shot_prompt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a0376cee46c008fc"}},{"code_sha256_prefix":"3346c670489dfe45","entry":"get_longest_solution","repo":"dsl-lab/aops","repo_kind":"official","path":"classify_aops.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/classify_aops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3346c670489dfe45"}},{"code_sha256_prefix":"1a400d0b49174a55","entry":"get_prompt_format","repo":"dsl-lab/aops","repo_kind":"official","path":"aops/few_shot_annotate.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/aops/few_shot_annotate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1a400d0b49174a55"}},{"code_sha256_prefix":"73f187d527e385d9","entry":"get_prompt_format","repo":"dsl-lab/aops","repo_kind":"official","path":"aops/few_shot_annotate_llama.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/aops/few_shot_annotate_llama.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73f187d527e385d9"}},{"code_sha256_prefix":"9b98b3ce3a455ce5","entry":"get_unparsed_batch","repo":"dsl-lab/aops","repo_kind":"official","path":"classify_aops.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/classify_aops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b98b3ce3a455ce5"}},{"code_sha256_prefix":"9f1f598046ebe8a3","entry":"get_unparsed_batch","repo":"dsl-lab/aops","repo_kind":"official","path":"parse_aops.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/parse_aops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f1f598046ebe8a3"}},{"code_sha256_prefix":"020f9cd4a09e546c","entry":"process_data","repo":"dsl-lab/aops","repo_kind":"official","path":"aops_crawler/clean_raw.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/aops_crawler/clean_raw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"020f9cd4a09e546c"}},{"code_sha256_prefix":"d681a47633b8316f","entry":"sanity_check","repo":"dsl-lab/aops","repo_kind":"official","path":"parse_aops.py","file_url":"https://github.com/dsl-lab/aops/blob/HEAD/parse_aops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d681a47633b8316f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}