{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-22299","title":"HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space","arxiv_id":"2509.22299","date":"2025-09-26","proceeding":"ICLR","authors":["Ke Li","Zheng Yang","Zhongbin Zhou","Feng Xue","Zhonglin Jiang","Wenxiao Wang"],"abstract":"Mixture-of-Experts (MoE) architectures in large language models (LLMs) deliver exceptional performance and reduced inference costs compared to dense LLMs. However, their large parameter counts result in prohibitive memory requirements, limiting practical deployment. While existing pruning methods primarily focus on expert-level pruning, this coarse granularity often leads to substantial accuracy degradation. In this work, we introduce HEAPr, a novel pruning algorithm that decomposes experts into smaller, indivisible atomic experts, enabling more precise and flexible atomic expert pruning. To measure the importance of each atomic expert, we leverage second-order information based on principles similar to the Optimal Brain Surgeon theory. To address the computational and storage challenges posed by second-order information, HEAPr exploits the inherent properties of atomic experts to transform the second-order information from expert parameters into that of atomic expert parameters, and further simplifies it to the second-order information of atomic expert outputs. This approach reduces the space complexity from $O(d^4)$, where $d$ is the model's dimensionality, to $O(d^2)$. HEAPr requires only two forward passes and one backward pass on a small calibration set to compute the importance of atomic experts. Extensive experiments on MoE models, including DeepSeek MoE and Qwen MoE family, demonstrate that HEAPr outperforms existing expert-level pruning methods across a wide range of pruning ratios and benchmarks. Specifically, HEAPr achieves nearly lossless compression at pruning ratios of 20% ~ 25% in most models, while also reducing FLOPs nearly by 20%. The code can be found at [https://github.com/LLIKKE/HEAPr](https://github.com/LLIKKE/HEAPr).","url_abs":"https://arxiv.org/abs/2509.22299","url_pdf":"https://arxiv.org/pdf/2509.22299","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2509.22299","atlas_url":"https://app.syntology.ai/?focus=2509.22299","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.22299"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/LLIKKE/HEAPr","reach":null}],"summary":{"ran":1,"ran_draft_wrong":3,"ran_honours":1,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"479b6a7d3bfaff16","entry":"IdentityExpertMLP","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"479b6a7d3bfaff16"}},{"code_sha256_prefix":"b0ad7d7f1847d19d","entry":"build_expert_sensitivity_hook","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0ad7d7f1847d19d"}},{"code_sha256_prefix":"22041aea171dc9f2","entry":"count_parameters","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22041aea171dc9f2"}},{"code_sha256_prefix":"fbbaa23d127ff313","entry":"find_named_layers","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbbaa23d127ff313"}},{"code_sha256_prefix":"e37512c7993b1db7","entry":"format_parameter_count","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e37512c7993b1db7"}},{"code_sha256_prefix":"d8fcb1e4d80cec40","entry":"clear_param_grads_backward_hook","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8fcb1e4d80cec40"}},{"code_sha256_prefix":"2e0b54372afbca7a","entry":"prune_expert_neurons_inplace","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e0b54372afbca7a"}},{"code_sha256_prefix":"ebf4fa2a7c794ca3","entry":"pruning_global","repo":"LLIKKE/HEAPr","repo_kind":"found_in_text","path":"pruning/pruning_global.py","file_url":"https://github.com/LLIKKE/HEAPr/blob/HEAD/pruning/pruning_global.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebf4fa2a7c794ca3"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}