{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/openwebmath-an-open-dataset-of-high-quality","title":"OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text","arxiv_id":"2310.06786","date":"2023-10-10","proceeding":null,"authors":["Keiran Paster","Marco Dos Santos","Zhangir Azerbayev","Jimmy Ba"],"abstract":"There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.","url_abs":"https://arxiv.org/abs/2310.06786v1","url_pdf":"https://arxiv.org/pdf/2310.06786v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"openwebmath-an-open-dataset-of-high-quality","repo_url":"https://github.com/keirp/OpenWebMath","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"openwebmath-an-open-dataset-of-high-quality","repo_url":"https://github.com/deepseek-ai/deepseek-math","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"palm","method_name":"PaLM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.06786","atlas_url":"https://app.syntology.ai/?focus=2310.06786","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.06786"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepseek-ai/deepseek-math","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keirp/OpenWebMath","reach":null}],"summary":{"ran_honours":1,"ran_violates":1,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"663f80e5ecdbafb8","entry":"count_latex_formulas","repo":"keirp/OpenWebMath","repo_kind":"official","path":"filtering/filter_dataset.py","file_url":"https://github.com/keirp/OpenWebMath/blob/HEAD/filtering/filter_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"663f80e5ecdbafb8"}},{"code_sha256_prefix":"527338d994af9eda","entry":"filter_data","repo":"keirp/OpenWebMath","repo_kind":"official","path":"filtering/filter_dataset.py","file_url":"https://github.com/keirp/OpenWebMath/blob/HEAD/filtering/filter_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"527338d994af9eda"}},{"code_sha256_prefix":"b0e2f9e9edc548d4","entry":"html_preprocessing","repo":"keirp/OpenWebMath","repo_kind":"official","path":"text_extraction/text_extract/extract.py","file_url":"https://github.com/keirp/OpenWebMath/blob/HEAD/text_extraction/text_extract/extract.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0e2f9e9edc548d4"}},{"code_sha256_prefix":"44b0a95807bd9a24","entry":"replace_tags","repo":"keirp/OpenWebMath","repo_kind":"official","path":"text_extraction/text_extract/extract.py","file_url":"https://github.com/keirp/OpenWebMath/blob/HEAD/text_extraction/text_extract/extract.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"44b0a95807bd9a24"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}