{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pruning-via-merging-compressing-llms-via","title":"Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging","arxiv_id":"2406.16330","date":"2024-06-24","proceeding":null,"authors":["Deyuan Liu","Zhanyue Qin","Hairu Wang","Zhao Yang","Zecheng Wang","Fangying Rong","Qingbin Liu","Yanchao Hao","Xi Chen","Cunhang Fan","Zhao Lv","Zhiying Tu","Dianhui Chu","Bo Li","Dianbo Sui"],"abstract":"While large language models (LLMs) excel in many domains, their complexity and scale challenge deployment in resource-limited environments. Current compression techniques, such as parameter pruning, often fail to effectively utilize the knowledge from pruned parameters. To address these challenges, we propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA), a novel approach that uses manifold learning and the Normalized Pairwise Information Bottleneck (NPIB) measure to merge similar layers, reducing model size while preserving essential performance. We evaluate MKA on multiple benchmark datasets and various LLMs. Our findings show that MKA not only preserves model performance but also achieves substantial compression ratios, outperforming traditional pruning methods. Moreover, when coupled with quantization, MKA delivers even greater compression. Specifically, on the MMLU dataset using the Llama3-8B model, MKA achieves a compression ratio of 43.75% with a minimal performance decrease of only 2.82\\%. The proposed MKA method offers a resource-efficient and performance-preserving model compression technique for LLMs.","url_abs":"https://arxiv.org/abs/2406.16330v2","url_pdf":"https://arxiv.org/pdf/2406.16330v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pruning-via-merging-compressing-llms-via","repo_url":"https://github.com/sempraety/pruning-via-merging","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.16330","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16330"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sempraety/pruning-via-merging","reach":null}],"summary":{"ran_draft_wrong":4,"ran_honours":3,"ran_fixture":3,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"9efeb174742eed9d","entry":"L2_distance_chunked","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9efeb174742eed9d"}},{"code_sha256_prefix":"0497b6ae852e62ab","entry":"adaptive_chunk_size","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0497b6ae852e62ab"}},{"code_sha256_prefix":"b283a13e98589a5b","entry":"compute_fusion_ratios","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b283a13e98589a5b"}},{"code_sha256_prefix":"0098461f3e6d61f2","entry":"compute_similarity_matrix_npib_global","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0098461f3e6d61f2"}},{"code_sha256_prefix":"bf95499b07bd5331","entry":"diffusionKernel","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf95499b07bd5331"}},{"code_sha256_prefix":"a8b5d262557261c3","entry":"entropy_estimator_knn","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8b5d262557261c3"}},{"code_sha256_prefix":"2a633a2060ebfeed","entry":"extract_layer_params","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a633a2060ebfeed"}},{"code_sha256_prefix":"e65eff7d9e1ab5f3","entry":"format_example","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e65eff7d9e1ab5f3"}},{"code_sha256_prefix":"fdf461465b0629eb","entry":"layer_fusion","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdf461465b0629eb"}},{"code_sha256_prefix":"ad08a638e2808f27","entry":"load_embeddings","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad08a638e2808f27"}},{"code_sha256_prefix":"7870ef64be87e3c2","entry":"clear_memory","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7870ef64be87e3c2"}},{"code_sha256_prefix":"7e604505814490b1","entry":"main","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e604505814490b1"}},{"code_sha256_prefix":"c0aae6f23602f51d","entry":"set_seed","repo":"sempraety/pruning-via-merging","repo_kind":"official","path":"pipeline.py","file_url":"https://github.com/sempraety/pruning-via-merging/blob/HEAD/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0aae6f23602f51d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}