{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-01650","title":"The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM","arxiv_id":"2510.01650","date":"2025-10-02","proceeding":"ICLR","authors":["Kwanhee Lee","Hyeondo Jang","Dongyeop Lee","Dan Alistarh","Namhoon Lee"],"abstract":"Neural network pruning is a promising technique to mitigate the excessive computational and memory requirements of large language models (LLMs). Despite its promise, however, progress in this area has diminished, as conventional methods are seemingly unable to surpass moderate sparsity levels (50-60%) without severely degrading model accuracy. This work breaks through the current impasse, presenting a principled and effective method called $\\texttt{Elsa}$, which achieves extreme sparsity levels of up to 90% while retaining high model fidelity. This is done by identifying several limitations in current practice, all of which can be traced back to their reliance on a surrogate objective formulation. $\\texttt{Elsa}$ tackles this issue directly and effectively via standard and well-established constrained optimization techniques based on ADMM. Our extensive experiments across a wide range of models and scales show that $\\texttt{Elsa}$ achieves substantial improvements over existing methods; e.g., it achieves 7.8$\\times$ less perplexity than the best existing method on LLaMA-2-7B at 90% sparsity. Moreover, we show that $\\texttt{Elsa}$ remains stable even at extreme sparsity (e.g., 95\\%), yielding up to $\\times$3.98 inference speedup and $\\times$7.80 memory compression over its dense counterpart. We also present $\\texttt{Elsa}_{-L}$, a quantized variant that scales to extremely large models (27B), and establish its theoretical convergence guarantees.These results highlight meaningful progress in advancing the frontier of LLM sparsity, while promising that significant opportunities for further advancement may remain in directions that have so far attracted limited exploration.","url_abs":"https://arxiv.org/abs/2510.01650","url_pdf":"https://arxiv.org/pdf/2510.01650","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.01650","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.01650"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/log-postech/elsa","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"6644b68819429769","entry":"find_layers","repo":"log-postech/elsa","repo_kind":"found_in_text","path":"lib/utils.py","file_url":"https://github.com/log-postech/elsa/blob/HEAD/lib/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6644b68819429769"}},{"code_sha256_prefix":"94fc9d5b353d22d8","entry":"get_dataset","repo":"log-postech/elsa","repo_kind":"found_in_text","path":"lib/data.py","file_url":"https://github.com/log-postech/elsa/blob/HEAD/lib/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94fc9d5b353d22d8"}},{"code_sha256_prefix":"e4eab462181ea68a","entry":"get_llm","repo":"log-postech/elsa","repo_kind":"found_in_text","path":"lib/utils.py","file_url":"https://github.com/log-postech/elsa/blob/HEAD/lib/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4eab462181ea68a"}},{"code_sha256_prefix":"e2c3c36c497f9fe0","entry":"get_loaders","repo":"log-postech/elsa","repo_kind":"found_in_text","path":"lib/data.py","file_url":"https://github.com/log-postech/elsa/blob/HEAD/lib/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2c3c36c497f9fe0"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}