{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/megamath-pushing-the-limits-of-open-math","title":"MegaMath: Pushing the Limits of Open Math Corpora","arxiv_id":"2504.02807","date":"2025-04-03","proceeding":null,"authors":["Fan Zhou","Zengzhi Wang","Nikhil Ranjan","Zhoujun Cheng","Liping Tang","Guowei He","Zhengzhong Liu","Eric P. Xing"],"abstract":"Mathematical reasoning is a cornerstone of human intelligence and a key benchmark for advanced capabilities in large language models (LLMs). However, the research community still lacks an open, large-scale, high-quality corpus tailored to the demands of math-centric LLM pre-training. We present MegaMath, an open dataset curated from diverse, math-focused sources through following practices: (1) Revisiting web data: We re-extracted mathematical documents from Common Crawl with math-oriented HTML optimizations, fasttext-based filtering and deduplication, all for acquiring higher-quality data on the Internet. (2) Recalling Math-related code data: We identified high quality math-related code from large code training corpus, Stack-V2, further enhancing data diversity. (3) Exploring Synthetic data: We synthesized QA-style text, math-related code, and interleaved text-code blocks from web data or code data. By integrating these strategies and validating their effectiveness through extensive ablations, MegaMath delivers 371B tokens with the largest quantity and top quality among existing open math pre-training datasets.","url_abs":"https://arxiv.org/abs/2504.02807v1","url_pdf":"https://arxiv.org/pdf/2504.02807v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"megamath-pushing-the-limits-of-open-math","repo_url":"https://github.com/llm360/megamath","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.02807","atlas_url":"https://app.syntology.ai/?focus=2504.02807","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.02807"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/llm360/megamath","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fe4e4053c2acea4a","entry":"get_list","repo":"llm360/megamath","repo_kind":"official","path":"web_pipeline/url_filtering/url_filter.py","file_url":"https://github.com/llm360/megamath/blob/HEAD/web_pipeline/url_filtering/url_filter.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe4e4053c2acea4a"}},{"code_sha256_prefix":"2c9e6858b942d667","entry":"html_table_to_markdown","repo":"llm360/megamath","repo_kind":"official","path":"web_pipeline/stage1_download_and_extract.py","file_url":"https://github.com/llm360/megamath/blob/HEAD/web_pipeline/stage1_download_and_extract.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c9e6858b942d667"}},{"code_sha256_prefix":"94cea2fe405ab4af","entry":"normalization","repo":"llm360/megamath","repo_kind":"official","path":"web_pipeline/utils/math_fasttext.py","file_url":"https://github.com/llm360/megamath/blob/HEAD/web_pipeline/utils/math_fasttext.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94cea2fe405ab4af"}},{"code_sha256_prefix":"a1a69da5e49cc696","entry":"normalize","repo":"llm360/megamath","repo_kind":"official","path":"web_pipeline/url_filtering/url_filter.py","file_url":"https://github.com/llm360/megamath/blob/HEAD/web_pipeline/url_filtering/url_filter.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1a69da5e49cc696"}},{"code_sha256_prefix":"32e16941520dc46f","entry":"parse_list","repo":"llm360/megamath","repo_kind":"official","path":"web_pipeline/url_filtering/url_filter.py","file_url":"https://github.com/llm360/megamath/blob/HEAD/web_pipeline/url_filtering/url_filter.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32e16941520dc46f"}},{"code_sha256_prefix":"9d1f40638d5bc810","entry":"preprocess_for_fasttext","repo":"llm360/megamath","repo_kind":"official","path":"web_pipeline/utils/math_fasttext.py","file_url":"https://github.com/llm360/megamath/blob/HEAD/web_pipeline/utils/math_fasttext.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d1f40638d5bc810"}},{"code_sha256_prefix":"9acbf779ceefa16c","entry":"download_from_cc","repo":"llm360/megamath","repo_kind":"official","path":"web_pipeline/utils/file_utils.py","file_url":"https://github.com/llm360/megamath/blob/HEAD/web_pipeline/utils/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9acbf779ceefa16c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}