{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynamic-data-mixing-maximizes-instruction","title":"Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts","arxiv_id":"2406.11256","date":"2024-06-17","proceeding":null,"authors":["Tong Zhu","Daize Dong","Xiaoye Qu","Jiacheng Ruan","Wenliang Chen","Yu Cheng"],"abstract":"Mixture-of-Experts (MoE) models have shown remarkable capability in instruction tuning, especially when the number of tasks scales. However, previous methods simply merge all training tasks (e.g. creative writing, coding, and mathematics) and apply fixed sampling weights, without considering the importance of different tasks as the model training state changes. In this way, the most helpful data cannot be effectively distinguished, leading to suboptimal model performance. To reduce the potential redundancies of datasets, we make the first attempt and propose a novel dynamic data mixture for MoE instruction tuning. Specifically, inspired by MoE's token routing preference, we build dataset-level representations and then capture the subtle differences among datasets. Finally, we propose to dynamically adjust the sampling weight of datasets by their inter-redundancies, thus maximizing global performance under a limited training budget. The experimental results on two MoE models demonstrate the effectiveness of our approach on both downstream knowledge \\& reasoning tasks and open-ended queries. Code and models are available at https://github.com/Spico197/MoE-SFT .","url_abs":"https://arxiv.org/abs/2406.11256v1","url_pdf":"https://arxiv.org/pdf/2406.11256v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamic-data-mixing-maximizes-instruction","repo_url":"https://github.com/spico197/moe-sft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.11256","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11256"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/spico197/moe-sft","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran":5,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"spico197/moe-sft","repo_kind":"official","path":"src/models/llama_moe/modeling_llama_moe.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/models/llama_moe/modeling_llama_moe.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"spico197/moe-sft","repo_kind":"official","path":"src/models/llama_moe/modeling_llama_moe.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/models/llama_moe/modeling_llama_moe.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"ac0ff7c1c276a33e","entry":"fault_tolerance_data_collator","repo":"spico197/moe-sft","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac0ff7c1c276a33e"}},{"code_sha256_prefix":"0298e66ff2a3e012","entry":"get_job_status","repo":"spico197/moe-sft","repo_kind":"official","path":"src/eval/listen.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/eval/listen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0298e66ff2a3e012"}},{"code_sha256_prefix":"f5031b2055ebdbd9","entry":"get_uniform_sampling_ratio","repo":"spico197/moe-sft","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5031b2055ebdbd9"}},{"code_sha256_prefix":"5f61aedc0be27ad7","entry":"job_is_runnnig","repo":"spico197/moe-sft","repo_kind":"official","path":"src/eval/listen.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/eval/listen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f61aedc0be27ad7"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"spico197/moe-sft","repo_kind":"official","path":"src/models/llama_moe/modeling_llama_moe.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/models/llama_moe/modeling_llama_moe.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"f9c665d8a29d201b","entry":"submit_job","repo":"spico197/moe-sft","repo_kind":"official","path":"src/eval/listen.py","file_url":"https://github.com/spico197/moe-sft/blob/HEAD/src/eval/listen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9c665d8a29d201b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}