{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-online-data-mixing-for-language","title":"Efficient Online Data Mixing For Language Model Pre-Training","arxiv_id":"2312.02406","date":"2023-12-05","proceeding":null,"authors":["Alon Albalak","Liangming Pan","Colin Raffel","William Yang Wang"],"abstract":"The data used to pretrain large language models has a decisive impact on a model's downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining. Existing data selection methods suffer from slow and computationally expensive processes, a problem amplified by the increasing size of models and of pretraining datasets. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together and determining sampling probabilities across entire groups. However, data mixing proportions are typically fixed before training and therefore cannot adapt to changing training dynamics. To address these limitations, we develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing. Based on multi-armed bandit algorithms, our online approach optimizes the data mixing proportions during training. Remarkably, our method trains a model that reaches the final perplexity of the next best method with 19\\% fewer training iterations, and improves performance on the 5-shot MMLU benchmark by 1.9% relative accuracy, while adding negligible wall-clock time during pretraining.","url_abs":"https://arxiv.org/abs/2312.02406v2","url_pdf":"https://arxiv.org/pdf/2312.02406v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-online-data-mixing-for-language","repo_url":"https://github.com/alon-albalak/online-data-mixing","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mmlu","task_name":"MMLU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02406","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02406"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alon-albalak/online-data-mixing","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":7},"by_repo_kind":{"listed":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"01edd35282158f9c","entry":"code","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"bigram_model.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/bigram_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"01edd35282158f9c"}},{"code_sha256_prefix":"32c8c6d61ee207b7","entry":"human_readable_flops","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"megatron/logging.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/megatron/logging.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"32c8c6d61ee207b7"}},{"code_sha256_prefix":"ec62b348d91f2d73","entry":"broadcast_data","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"megatron/mpu/data.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/megatron/mpu/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ec62b348d91f2d73"}},{"code_sha256_prefix":"bdf28f9492a770dc","entry":"data_file_path","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"bigram_model.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/bigram_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bdf28f9492a770dc"}},{"code_sha256_prefix":"750d7ac5c014981c","entry":"get_docs","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"configs/gen_docs.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/configs/gen_docs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"750d7ac5c014981c"}},{"code_sha256_prefix":"4f686edc1148a835","entry":"get_flops","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"megatron/logging.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/megatron/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4f686edc1148a835"}},{"code_sha256_prefix":"8b1a74a6db32c163","entry":"index_file_path","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"bigram_model.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/bigram_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8b1a74a6db32c163"}},{"code_sha256_prefix":"50c9c165cf801f59","entry":"pairwise","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"configs/gen_docs.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/configs/gen_docs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"50c9c165cf801f59"}},{"code_sha256_prefix":"64dd872a8ac47294","entry":"to_md","repo":"alon-albalak/online-data-mixing","repo_kind":"listed","path":"configs/gen_docs.py","file_url":"https://github.com/alon-albalak/online-data-mixing/blob/HEAD/configs/gen_docs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"64dd872a8ac47294"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}