{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/higher-layers-need-more-lora-experts","title":"Higher Layers Need More LoRA Experts","arxiv_id":"2402.08562","date":"2024-02-13","proceeding":null,"authors":["Chongyang Gao","Kezhen Chen","Jinmeng Rao","Baochen Sun","Ruibo Liu","Daiyi Peng","Yawen Zhang","Xiaoyuan Guo","Jie Yang","VS Subrahmanian"],"abstract":"Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite promising results, research on improving the efficiency of LoRA with MoE is still in its early stages. Recent studies have shown that experts in the MoE architecture have different strengths and also exhibit some redundancy. Does this statement also apply to parameter-efficient MoE? In this paper, we introduce a novel parameter-efficient MoE method, \\textit{\\textbf{M}oE-L\\textbf{o}RA with \\textbf{L}ayer-wise Expert \\textbf{A}llocation (MoLA)} for Transformer-based models, where each model layer has the flexibility to employ a varying number of LoRA experts. We investigate several architectures with varying layer-wise expert configurations. Experiments on six well-known NLP and commonsense QA benchmarks demonstrate that MoLA achieves equal or superior performance compared to all baselines. We find that allocating more LoRA experts to higher layers further enhances the effectiveness of models with a certain number of experts in total. With much fewer parameters, this allocation strategy outperforms the setting with the same number of experts in every layer. This work can be widely used as a plug-and-play parameter-efficient tuning approach for various applications. The code is available at https://github.com/GCYZSL/MoLA.","url_abs":"https://arxiv.org/abs/2402.08562v1","url_pdf":"https://arxiv.org/pdf/2402.08562v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"higher-layers-need-more-lora-experts","repo_url":"https://github.com/gcyzsl/mola","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.08562","atlas_url":"https://app.syntology.ai/?focus=2402.08562","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.08562"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gcyzsl/mola","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":4},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"7b4d35d5ca974ddd","entry":"detect_nan_inf","repo":"gcyzsl/mola","repo_kind":"official","path":"src/mola_modeling_llama_hacked.py","file_url":"https://github.com/gcyzsl/mola/blob/HEAD/src/mola_modeling_llama_hacked.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b4d35d5ca974ddd"}},{"code_sha256_prefix":"30a82d453ae174bf","entry":"get_answer","repo":"gcyzsl/mola","repo_kind":"official","path":"preparation_scienceqa_data.py","file_url":"https://github.com/gcyzsl/mola/blob/HEAD/preparation_scienceqa_data.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30a82d453ae174bf"}},{"code_sha256_prefix":"7a8dffc83d89cedd","entry":"get_choice_text","repo":"gcyzsl/mola","repo_kind":"official","path":"preparation_scienceqa_data.py","file_url":"https://github.com/gcyzsl/mola/blob/HEAD/preparation_scienceqa_data.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a8dffc83d89cedd"}},{"code_sha256_prefix":"85f084f6054a421e","entry":"get_context_text","repo":"gcyzsl/mola","repo_kind":"official","path":"preparation_scienceqa_data.py","file_url":"https://github.com/gcyzsl/mola/blob/HEAD/preparation_scienceqa_data.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85f084f6054a421e"}},{"code_sha256_prefix":"94e2e2c1c3336f07","entry":"load_balancing_loss_func","repo":"gcyzsl/mola","repo_kind":"official","path":"src/mola_modeling_llama_hacked.py","file_url":"https://github.com/gcyzsl/mola/blob/HEAD/src/mola_modeling_llama_hacked.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94e2e2c1c3336f07"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"gcyzsl/mola","repo_kind":"official","path":"src/mola_modeling_llama_hacked.py","file_url":"https://github.com/gcyzsl/mola/blob/HEAD/src/mola_modeling_llama_hacked.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}