{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/outlier-weighed-layerwise-sparsity-owl-a","title":"Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity","arxiv_id":"2310.05175","date":"2023-10-08","proceeding":null,"authors":["Lu Yin","You Wu","Zhenyu Zhang","Cheng-Yu Hsieh","Yaqing Wang","Yiling Jia","Gen Li","Ajay Jaiswal","Mykola Pechenizkiy","Yi Liang","Michael Bendersky","Zhangyang Wang","Shiwei Liu"],"abstract":"Large Language Models (LLMs), renowned for their remarkable performance across diverse domains, present a challenge when it comes to practical deployment due to their colossal model size. In response to this challenge, efforts have been directed toward the application of traditional network pruning techniques to LLMs, uncovering a massive number of parameters that can be pruned in one-shot without hurting performance. Prevailing LLM pruning strategies have consistently adhered to the practice of uniformly pruning all layers at equivalent sparsity, resulting in robust performance. However, this observation stands in contrast to the prevailing trends observed in the field of vision models, where non-uniform layerwise sparsity typically yields stronger results. To understand the underlying reasons for this disparity, we conduct a comprehensive study and discover a strong correlation with the emergence of activation outliers in LLMs. Inspired by this finding, we introduce a novel LLM pruning methodology that incorporates a tailored set of non-uniform layerwise sparsity ratios, termed as Outlier Weighed Layerwise sparsity (OWL). The sparsity ratio of OWL is proportional to the outlier ratio observed within each layer, facilitating a more effective alignment between layerwise weight sparsity and outlier ratios. Our empirical evaluation, conducted across the LLaMA-V1 family and OPT, spanning various benchmarks, demonstrates the distinct advantages offered by OWL over previous methods. For instance, OWL exhibits a remarkable performance gain, surpassing the state-of-the-art Wanda and SparseGPT by 61.22 and 6.80 perplexity at a high sparsity level of 70%, respectively, while delivering 2.6x end-to-end inference speed-up in the DeepSparse inference engine. Codes are available at https://github.com/luuyin/OWL.","url_abs":"https://arxiv.org/abs/2310.05175v3","url_pdf":"https://arxiv.org/pdf/2310.05175v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"outlier-weighed-layerwise-sparsity-owl-a","repo_url":"https://github.com/luuyin/owl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"network-pruning","task_name":"Network Pruning"}],"methods":[{"method_slug":"opt","method_name":"OPT"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.05175","atlas_url":"https://app.syntology.ai/?focus=2310.05175","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05175"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luuyin/owl","reach":null}],"summary":{"ran":1,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c496051510743116","entry":"WrappedGPT","repo":"luuyin/owl","repo_kind":"official","path":"lib/prune_all.py","file_url":"https://github.com/luuyin/owl/blob/HEAD/lib/prune_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c496051510743116"}},{"code_sha256_prefix":"850653feea44c31d","entry":"check_outlier_mean","repo":"luuyin/owl","repo_kind":"official","path":"lib/prune_all.py","file_url":"https://github.com/luuyin/owl/blob/HEAD/lib/prune_all.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"850653feea44c31d"}},{"code_sha256_prefix":"2d4b7e939c6fa46b","entry":"get_loaders","repo":"luuyin/owl","repo_kind":"official","path":"lib/prune_all.py","file_url":"https://github.com/luuyin/owl/blob/HEAD/lib/prune_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d4b7e939c6fa46b"}},{"code_sha256_prefix":"298323fc64be86fa","entry":"prepare_calibration_input","repo":"luuyin/owl","repo_kind":"official","path":"lib/prune_all.py","file_url":"https://github.com/luuyin/owl/blob/HEAD/lib/prune_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"298323fc64be86fa"}},{"code_sha256_prefix":"b604c7050dd74130","entry":"prepare_calibration_input_opt","repo":"luuyin/owl","repo_kind":"official","path":"lib/prune_all.py","file_url":"https://github.com/luuyin/owl/blob/HEAD/lib/prune_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b604c7050dd74130"}},{"code_sha256_prefix":"9f8e6480ac44436c","entry":"prune_wanda_outlier_structure_special","repo":"luuyin/owl","repo_kind":"official","path":"lib/prune_all.py","file_url":"https://github.com/luuyin/owl/blob/HEAD/lib/prune_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f8e6480ac44436c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}