{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-mixing-laws-optimizing-data-mixtures-by","title":"Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance","arxiv_id":"2403.16952","date":"2024-03-25","proceeding":null,"authors":["Jiasheng Ye","Peiju Liu","Tianxiang Sun","Yunhua Zhou","Jun Zhan","Xipeng Qiu"],"abstract":"Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before actual runs, thus guiding the selection of an ideal data mixture. Furthermore, we propose nested use of the scaling laws of training steps, model sizes, and our data mixing law to enable predicting the performance of large models trained on massive data under various mixtures with only small-scale training. Moreover, experimental results verify that our method effectively optimizes the training mixture of a 1B model trained for 100B tokens in RedPajama, reaching a performance comparable to the one trained for 48% more steps on the default mixture. Extending the application of data mixing laws to continual training accurately predicts the critical mixture proportion that avoids catastrophic forgetting and outlooks the potential for dynamic data schedules","url_abs":"https://arxiv.org/abs/2403.16952v1","url_pdf":"https://arxiv.org/pdf/2403.16952v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-mixing-laws-optimizing-data-mixtures-by","repo_url":"https://github.com/yegcjs/mixinglaws","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.16952","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.16952"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yegcjs/mixinglaws","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":5},"by_repo_kind":{"official":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"633fdf04a3299a5a","entry":"dfs","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"pipeline/find_opt.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/pipeline/find_opt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"633fdf04a3299a5a"}},{"code_sha256_prefix":"43bd510fe401e6e2","entry":"mixture_law_","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"pipeline/utils.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43bd510fe401e6e2"}},{"code_sha256_prefix":"a1130b34b321dd67","entry":"power_law_","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"pipeline/utils.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1130b34b321dd67"}},{"code_sha256_prefix":"bac1e76d6b48db48","entry":"power_law_share_alpha_","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"pipeline/utils.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bac1e76d6b48db48"}},{"code_sha256_prefix":"cc3ad484cd8dd56c","entry":"calculate_r_squared","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"law.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/law.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc3ad484cd8dd56c"}},{"code_sha256_prefix":"da653fe399fe00af","entry":"fit_regressor","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"pipeline/mixlaw.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/pipeline/mixlaw.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da653fe399fe00af"}},{"code_sha256_prefix":"ae58a6abc7dd4a7e","entry":"fit_scaling_laws","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"law.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/law.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae58a6abc7dd4a7e"}},{"code_sha256_prefix":"199c1d9654de273d","entry":"load_loss","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"pipeline/get_loss.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/pipeline/get_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"199c1d9654de273d"}},{"code_sha256_prefix":"0f605b00b7a97819","entry":"predict_losses","repo":"yegcjs/mixinglaws","repo_kind":"official","path":"pipeline/find_opt.py","file_url":"https://github.com/yegcjs/mixinglaws/blob/HEAD/pipeline/find_opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f605b00b7a97819"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}