{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fineweb2-one-pipeline-to-scale-them-all","title":"FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language","arxiv_id":"2506.20920","date":"2025-06-26","proceeding":null,"authors":["Guilherme Penedo","Hynek Kydlíček","Vinko Sabolčec","Bettina Messmer","Negar Foroutan","Amir Hossein Kargaran","Colin Raffel","Martin Jaggi","Leandro von Werra","Thomas Wolf"],"abstract":"Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-training datasets has seen substantial recent progress, training performant multilingual LLMs remains a challenge, in large part due to the inherent difficulty of tailoring filtering and deduplication pipelines to a large number of languages. In this work, we introduce a new pre-training dataset curation pipeline based on FineWeb that can be automatically adapted to support any language. We extensively ablate our pipeline design choices on a set of nine diverse languages, guided by a set of meaningful and informative evaluation tasks that were chosen through a novel selection process based on measurable criteria. Ultimately, we show that our pipeline can be used to create non-English corpora that produce more performant models than prior datasets. We additionally introduce a straightforward and principled approach to rebalance datasets that takes into consideration both duplication count and quality, providing an additional performance uplift. Finally, we scale our pipeline to over 1000 languages using almost 100 Common Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document) multilingual dataset which we release along with our pipeline, training, and evaluation codebases.","url_abs":"https://arxiv.org/abs/2506.20920v1","url_pdf":"https://arxiv.org/pdf/2506.20920v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fineweb2-one-pipeline-to-scale-them-all","repo_url":"https://github.com/huggingface/fineweb-2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.20920","atlas_url":"https://app.syntology.ai/?focus=2506.20920","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.20920"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/fineweb-2","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e3c44d1a1854c3fe","entry":"checkpoint_exists","repo":"huggingface/fineweb-2","repo_kind":"official","path":"ablations/evaluation/launch_evals.py","file_url":"https://github.com/huggingface/fineweb-2/blob/HEAD/ablations/evaluation/launch_evals.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3c44d1a1854c3fe"}},{"code_sha256_prefix":"25e77cb73f28dda5","entry":"parse_date","repo":"huggingface/fineweb-2","repo_kind":"official","path":"ablations/evaluation/launch_evals.py","file_url":"https://github.com/huggingface/fineweb-2/blob/HEAD/ablations/evaluation/launch_evals.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25e77cb73f28dda5"}},{"code_sha256_prefix":"ab9a508aaade952b","entry":"launch_slurm_job","repo":"huggingface/fineweb-2","repo_kind":"official","path":"ablations/evaluation/launch_evals.py","file_url":"https://github.com/huggingface/fineweb-2/blob/HEAD/ablations/evaluation/launch_evals.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab9a508aaade952b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}