{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/regmix-data-mixture-as-regression-for","title":"RegMix: Data Mixture as Regression for Language Model Pre-training","arxiv_id":"2407.01492","date":"2024-07-01","proceeding":null,"authors":["Qian Liu","Xiaosen Zheng","Niklas Muennighoff","Guangtao Zeng","Longxu Dou","Tianyu Pang","Jing Jiang","Min Lin"],"abstract":"The data mixture for large language model pre-training significantly impacts performance, yet how to determine an effective mixture remains unclear. We propose RegMix to automatically identify a high-performing data mixture by formulating it as a regression task. RegMix trains many small models on diverse data mixtures, uses regression to predict performance of unseen mixtures, and applies the best predicted mixture to train a large-scale model with orders of magnitude more compute. To empirically validate RegMix, we train 512 models with 1M parameters for 1B tokens to fit the regression model and predict the best data mixture. Using this mixture we train a 1B parameter model for 25B tokens (i.e. 1000x larger and 25x longer) which we find performs best among 64 candidate 1B parameter models with other mixtures. Furthermore, RegMix consistently outperforms human selection in experiments involving models up to 7B models trained on 100B tokens, while matching or exceeding DoReMi using just 10% of the computational resources. Our experiments also show that (1) Data mixtures significantly impact performance; (2) Web corpora rather than data perceived as high-quality like Wikipedia have the strongest positive correlation with downstream performance; (3) Domains interact in complex ways often contradicting common sense, thus automatic approaches like RegMix are needed; (4) Data mixture effects transcend scaling laws. Our code is available at https://github.com/sail-sg/regmix.","url_abs":"https://arxiv.org/abs/2407.01492v2","url_pdf":"https://arxiv.org/pdf/2407.01492v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"regmix-data-mixture-as-regression-for","repo_url":"https://github.com/sail-sg/regmix","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.01492","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.01492"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sail-sg/regmix","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":3},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"76dd60969be5e35b","entry":"generate_train_group","repo":"sail-sg/regmix","repo_kind":"official","path":"mixture_config/synthesize_mixture.py","file_url":"https://github.com/sail-sg/regmix/blob/HEAD/mixture_config/synthesize_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76dd60969be5e35b"}},{"code_sha256_prefix":"08d4762a51ea3e8f","entry":"generate_valid_group","repo":"sail-sg/regmix","repo_kind":"official","path":"mixture_config/synthesize_mixture.py","file_url":"https://github.com/sail-sg/regmix/blob/HEAD/mixture_config/synthesize_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08d4762a51ea3e8f"}},{"code_sha256_prefix":"3ac3f81e59a054e5","entry":"generate_weights_dirichlet","repo":"sail-sg/regmix","repo_kind":"official","path":"mixture_config/synthesize_mixture.py","file_url":"https://github.com/sail-sg/regmix/blob/HEAD/mixture_config/synthesize_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ac3f81e59a054e5"}},{"code_sha256_prefix":"78bd76c7783c9e22","entry":"generate_weights_dirichlet","repo":"sail-sg/regmix","repo_kind":"official","path":"mixture_config/synthesize_mixture.py","file_url":"https://github.com/sail-sg/regmix/blob/HEAD/mixture_config/synthesize_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78bd76c7783c9e22"}},{"code_sha256_prefix":"e10eeb439e12c9a9","entry":"sort_and_deduplicate","repo":"sail-sg/regmix","repo_kind":"official","path":"mixture_config/synthesize_mixture.py","file_url":"https://github.com/sail-sg/regmix/blob/HEAD/mixture_config/synthesize_mixture.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e10eeb439e12c9a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}