{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llama-moe-v2-exploring-sparsity-of-llama-from","title":"LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training","arxiv_id":"2411.15708","date":"2024-11-24","proceeding":null,"authors":["Xiaoye Qu","Daize Dong","Xuyang Hu","Tong Zhu","Weigao Sun","Yu Cheng"],"abstract":"Recently, inspired by the concept of sparsity, Mixture-of-Experts (MoE) models have gained increasing popularity for scaling model size while keeping the number of activated parameters constant. In this study, we thoroughly investigate the sparsity of the dense LLaMA model by constructing MoE for both the attention (i.e., Attention MoE) and MLP (i.e., MLP MoE) modules in the transformer blocks. Specifically, we investigate different expert construction methods and granularities under the same activation conditions to analyze the impact of sparsifying the model. Additionally, to comprehensively evaluate the model's capabilities across various domains (e.g., conversation, code, math) after sparsification, we apply sparsity to the instructed large language models (LLMs) and construct instructed MoE models. To counteract the performance degradation resulting from increased sparsity, we design a two-stage post-training strategy to enhance model performance. Experiments on the LLaMA3 model demonstrate the potential effectiveness of this approach for future developments of instructed MoE models. The source codes and models are available at: \\url{https://github.com/OpenSparseLLMs/LLaMA-MoE-v2}.","url_abs":"https://arxiv.org/abs/2411.15708v1","url_pdf":"https://arxiv.org/pdf/2411.15708v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llama-moe-v2-exploring-sparsity-of-llama-from","repo_url":"https://github.com/opensparsellms/llama-moe-v2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"moe","method_name":"MoE"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.15708","atlas_url":"https://app.syntology.ai/?focus=2411.15708","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.15708"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opensparsellms/llama-moe-v2","reach":null}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"166e224261b42266","entry":"fault_tolerance_data_collator","repo":"opensparsellms/llama-moe-v2","repo_kind":"official","path":"smoe/entrypoint/sft/train_sft_llama3_nopad.py","file_url":"https://github.com/opensparsellms/llama-moe-v2/blob/HEAD/smoe/entrypoint/sft/train_sft_llama3_nopad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"166e224261b42266"}},{"code_sha256_prefix":"ca021b7b8acf4e5a","entry":"simple_fault_tolerance_data_collator","repo":"opensparsellms/llama-moe-v2","repo_kind":"official","path":"smoe/entrypoint/sft/train_sft_llama3_nopad.py","file_url":"https://github.com/opensparsellms/llama-moe-v2/blob/HEAD/smoe/entrypoint/sft/train_sft_llama3_nopad.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca021b7b8acf4e5a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}