{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/olica-efficient-structured-pruning-of-large","title":"Olica: Efficient Structured Pruning of Large Language Models without Retraining","arxiv_id":"2506.08436","date":"2025-06-10","proceeding":null,"authors":["Jiujun He","Huazhen Lin"],"abstract":"Most existing structured pruning methods for Large Language Models (LLMs) require substantial computational and data resources for retraining to reestablish the corrupted correlations, making them prohibitively expensive. To address this, we propose a pruning framework for LLMs called Orthogonal decomposition and Linear Calibration (Olica), which eliminates the need for retraining. A key observation is that the multi-head attention (MHA) layer depends on two types of matrix products. By treating these matrix products as unified entities and applying principal component analysis (PCA), we extract the most important information to compress LLMs without sacrificing accuracy or disrupting their original structure. Consequently, retraining becomes unnecessary. A fast decomposition method is devised, reducing the complexity of PCA by a factor of the square of the number of attention heads. Additionally, to mitigate error accumulation problem caused by pruning the feed-forward network (FFN) layer, we introduce a linear calibration method to reconstruct the residual errors of pruned layers using low-rank matrices. By leveraging singular value decomposition (SVD) on the solution of the least-squares problem, these matrices are obtained without requiring retraining. Extensive experiments show that the proposed Olica is efficient in terms of data usage, GPU memory, and running time, while delivering superior performance across multiple benchmarks.","url_abs":"https://arxiv.org/abs/2506.08436v1","url_pdf":"https://arxiv.org/pdf/2506.08436v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"olica-efficient-structured-pruning-of-large","repo_url":"https://github.com/bettertmrr/llm-olica","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"pca","method_name":"PCA"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.08436","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.08436"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bettertmrr/llm-olica","reach":null}],"summary":{"ran":2,"ran_draft_wrong":1,"ran_honours":1,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"3b27642c8e33d464","entry":"CustomizedMLP","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b27642c8e33d464"}},{"code_sha256_prefix":"a1c8ad2d14601495","entry":"WrappedGPT","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1c8ad2d14601495"}},{"code_sha256_prefix":"8d18b53fd9470446","entry":"solve","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d18b53fd9470446"}},{"code_sha256_prefix":"9585b3f4219dd186","entry":"sr_allocation","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9585b3f4219dd186"}},{"code_sha256_prefix":"9fdb5bf5bea2f93d","entry":"SVDLinearForWidth","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fdb5bf5bea2f93d"}},{"code_sha256_prefix":"f42a0e10cdf30d57","entry":"fast_OND","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f42a0e10cdf30d57"}},{"code_sha256_prefix":"b174b3c5b1ffc583","entry":"forward","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b174b3c5b1ffc583"}},{"code_sha256_prefix":"387f9d5a5e0b0215","entry":"olica_pruning","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"387f9d5a5e0b0215"}},{"code_sha256_prefix":"22bc9f744c5dfe8f","entry":"pruning","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22bc9f744c5dfe8f"}},{"code_sha256_prefix":"e87aa42e51718185","entry":"thinner_mlp","repo":"bettertmrr/llm-olica","repo_kind":"official","path":"model_pruning.py","file_url":"https://github.com/bettertmrr/llm-olica/blob/HEAD/model_pruning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e87aa42e51718185"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}