{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/disp-llm-dimension-independent-structural","title":"DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models","arxiv_id":"2410.11988","date":"2024-10-15","proceeding":null,"authors":["Shangqian Gao","Chi-Heng Lin","Ting Hua","Tang Zheng","Yilin Shen","Hongxia Jin","Yen-Chang Hsu"],"abstract":"Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource-limited devices. Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural dependence along the embedding dimension. Our dimension-independent structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, we facilitate each block to possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate our method on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-art methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.","url_abs":"https://arxiv.org/abs/2410.11988v2","url_pdf":"https://arxiv.org/pdf/2410.11988v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"disp-llm-dimension-independent-structural","repo_url":"https://github.com/ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"opt","method_name":"OPT"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.11988","atlas_url":"https://app.syntology.ai/?focus=2410.11988","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.11988"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":4,"ran":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"b5e7666a7a4fda7e","entry":"gumbel_sigmoid_sample","repo":"ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning","repo_kind":"official","path":"pruning/hypernetwork.py","file_url":"https://github.com/ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning/blob/HEAD/pruning/hypernetwork.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5e7666a7a4fda7e"}},{"code_sha256_prefix":"4be412e5955fd46f","entry":"hard_sample","repo":"ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning","repo_kind":"official","path":"pruning/hypernetwork.py","file_url":"https://github.com/ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning/blob/HEAD/pruning/hypernetwork.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4be412e5955fd46f"}},{"code_sha256_prefix":"fa507755331041e1","entry":"hypernetwork","repo":"ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning","repo_kind":"official","path":"pruning/hypernetwork.py","file_url":"https://github.com/ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning/blob/HEAD/pruning/hypernetwork.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa507755331041e1"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"6fa35ad4a5d7f4c1","entry":"round_to_block_size","repo":"ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning","repo_kind":"official","path":"train_hypernetwork.py","file_url":"https://github.com/ZhengaoLi/DISP-LLM-Dimension-Independent-Structural-Pruning/blob/HEAD/train_hypernetwork.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6fa35ad4a5d7f4c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}