{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ntk-approximating-mlp-fusion-for-efficient","title":"MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models","arxiv_id":"2307.08941","date":"2023-07-18","proceeding":null,"authors":["Mengting Ai","Tianxin Wei","Yifan Chen","Zeming Guo","Jingrui He"],"abstract":"Fine-tuning a pre-trained language model (PLM) emerges as the predominant strategy in many natural language processing applications. However, this process is known to be expensive, especially on edge devices with low computing power. While general approaches (e.g. quantization and distillation) have been widely studied to reduce the compute/memory of PLM fine-tuning, one-shot compression techniques specifically designed for fine-tuning remain largely unexplored. In this paper, we investigate the neural tangent kernel (NTK)--which reveals the gradient descent dynamics of neural networks--of the multilayer perceptrons (MLP) modules in a PLM and propose to coin a lightweight PLM through NTK-approximating MLP fusion. By incorporating NTK into the compression process, MLP Fusion not only preserves the original model's output but also maintains its training dynamics. To achieve this, we reconsider the MLP as a bundle of sub-MLPs and cluster them into a given number of centroids, which can then be restored as a compressed MLP and surprisingly well approximate the NTK of the original PLM. Our approach is applicable to both standard MLP modules and Mixture-of-Experts (MoE) modules in PLMs, demonstrating its scalability and versatility. Additionally, we provide theoretical derivations to demonstrate how the proposed compression preserves the NTK. Extensive experiments of PLM fine-tuning on both natural language understanding and generation tasks are provided to verify the effectiveness of MLP fusion. Our code is available at https://github.com/weitianxin/MLP_Fusion.","url_abs":"https://arxiv.org/abs/2307.08941v3","url_pdf":"https://arxiv.org/pdf/2307.08941v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ntk-approximating-mlp-fusion-for-efficient","repo_url":"https://github.com/weitianxin/mlp_fusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"ntk","method_name":"NTK"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.08941","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.08941"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/weitianxin/MLP_Fusion","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/weitianxin/mlp_fusion","reach":{"status":"ok"}}],"summary":{"ran":9,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"8e65852945fc41b9","entry":"Linear_sketch","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e65852945fc41b9"}},{"code_sha256_prefix":"51c5a7fc7ae52036","entry":"Linear_svd","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51c5a7fc7ae52036"}},{"code_sha256_prefix":"0081b17668e7a074","entry":"MMD_model","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0081b17668e7a074"}},{"code_sha256_prefix":"77ecf773c5c9651a","entry":"ScaleLayer","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77ecf773c5c9651a"}},{"code_sha256_prefix":"905073406c036f74","entry":"SwitchTransformersConfig","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"905073406c036f74"}},{"code_sha256_prefix":"38f46ef4e03fee1c","entry":"create_position_ids_from_input_ids","repo":"weitianxin/MLP_Fusion","repo_kind":"official","path":"models/roberta/modeling_roberta.py","file_url":"https://github.com/weitianxin/MLP_Fusion/blob/HEAD/models/roberta/modeling_roberta.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38f46ef4e03fee1c"}},{"code_sha256_prefix":"6697ea7953dcb4c6","entry":"create_position_ids_from_input_ids","repo":"weitianxin/MLP_Fusion","repo_kind":"official","path":"models/roberta/modeling_flax_roberta.py","file_url":"https://github.com/weitianxin/MLP_Fusion/blob/HEAD/models/roberta/modeling_flax_roberta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6697ea7953dcb4c6"}},{"code_sha256_prefix":"9fea21a34c180fb0","entry":"initial_val_plus_disturbance","repo":"weitianxin/MLP_Fusion","repo_kind":"official","path":"nlg/adapters/low_rank_layer.py","file_url":"https://github.com/weitianxin/MLP_Fusion/blob/HEAD/nlg/adapters/low_rank_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fea21a34c180fb0"}},{"code_sha256_prefix":"695eaa24be6f2a1d","entry":"top_filtering","repo":"weitianxin/MLP_Fusion","repo_kind":"official","path":"nlg/evaluate.py","file_url":"https://github.com/weitianxin/MLP_Fusion/blob/HEAD/nlg/evaluate.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"695eaa24be6f2a1d"}},{"code_sha256_prefix":"0178efd5619a35d7","entry":"truncated_gaussian","repo":"weitianxin/MLP_Fusion","repo_kind":"official","path":"nlg/adapters/low_rank_layer.py","file_url":"https://github.com/weitianxin/MLP_Fusion/blob/HEAD/nlg/adapters/low_rank_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0178efd5619a35d7"}},{"code_sha256_prefix":"2005938ffe99509c","entry":"SwitchTransformersDenseActDense","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2005938ffe99509c"}},{"code_sha256_prefix":"18725727fe1cf6cb","entry":"SwitchTransformersSparseMLP","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18725727fe1cf6cb"}},{"code_sha256_prefix":"a04df7ea135e2987","entry":"SwitchTransformersTop1Router","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a04df7ea135e2987"}},{"code_sha256_prefix":"7c85bc322e1c67a7","entry":"structure_prune","repo":"weitianxin/mlp_fusion","repo_kind":"official","path":"switch/switch_transformers/modeling_switch_transformers.py","file_url":"https://github.com/weitianxin/mlp_fusion/blob/HEAD/switch/switch_transformers/modeling_switch_transformers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c85bc322e1c67a7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}