{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-super-weight-in-large-language-models","title":"The Super Weight in Large Language Models","arxiv_id":"2411.07191","date":"2024-11-11","proceeding":null,"authors":["Mengxia Yu","De Wang","Qi Shan","Colorado Reed","Alvin Wan"],"abstract":"Recent works have shown a surprising result: a small fraction of Large Language Model (LLM) parameter outliers are disproportionately important to the quality of the model. LLMs contain billions of parameters, so these small fractions, such as 0.01%, translate to hundreds of thousands of parameters. In this work, we present an even more surprising finding: Pruning as few as a single parameter can destroy an LLM's ability to generate text -- increasing perplexity by 3 orders of magnitude and reducing zero-shot accuracy to guessing. We propose a data-free method for identifying such parameters, termed super weights, using a single forward pass through the model. We additionally find that these super weights induce correspondingly rare and large activation outliers, termed super activations. When preserved with high precision, super activations can improve simple round-to-nearest quantization to become competitive with state-of-the-art methods. For weight quantization, we similarly find that by preserving the super weight and clipping other weight outliers, round-to-nearest quantization can scale to much larger block sizes than previously considered. To facilitate further research into super weights, we provide an index of super weight coordinates for common, openly available LLMs.","url_abs":"https://arxiv.org/abs/2411.07191v1","url_pdf":"https://arxiv.org/pdf/2411.07191v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-super-weight-in-large-language-models","repo_url":"https://github.com/mengxiayu/llmsuperweight","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.07191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.07191"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mengxiayu/llmsuperweight","reach":{"status":"ok"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"31d554f9c08a3997","entry":"check_module_hidden_states","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31d554f9c08a3997"}},{"code_sha256_prefix":"81b0669bd49b2880","entry":"dequantize_4bit_to_fp16","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"outliers/functional/quantization.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/outliers/functional/quantization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81b0669bd49b2880"}},{"code_sha256_prefix":"7b240c035c3e9665","entry":"get_layer_number","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"outliers/model.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/outliers/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b240c035c3e9665"}},{"code_sha256_prefix":"d2a93c35878e7797","entry":"get_weight_type","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"outliers/model.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/outliers/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2a93c35878e7797"}},{"code_sha256_prefix":"548d5eb845778c73","entry":"pack_4bit_to_int8","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"outliers/model.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/outliers/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"548d5eb845778c73"}},{"code_sha256_prefix":"54d74d789750136d","entry":"plot_token_probs","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54d74d789750136d"}},{"code_sha256_prefix":"8d67fa05e4442337","entry":"record_SO","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d67fa05e4442337"}},{"code_sha256_prefix":"c820570c06da4b08","entry":"unpack_int8_to_4bit","repo":"mengxiayu/llmsuperweight","repo_kind":"official","path":"outliers/functional/quantization.py","file_url":"https://github.com/mengxiayu/llmsuperweight/blob/HEAD/outliers/functional/quantization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c820570c06da4b08"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}