{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pb-llm-partially-binarized-large-language","title":"PB-LLM: Partially Binarized Large Language Models","arxiv_id":"2310.00034","date":"2023-09-29","proceeding":null,"authors":["Yuzhang Shang","Zhihang Yuan","Qiang Wu","Zhen Dong"],"abstract":"This paper explores network binarization, a radical form of quantization, compressing model weights to a single bit, specifically for Large Language Models (LLMs) compression. Due to previous binarization methods collapsing LLMs, we propose a novel approach, Partially-Binarized LLM (PB-LLM), which can achieve extreme low-bit quantization while maintaining the linguistic reasoning capacity of quantized LLMs. Specifically, our exploration first uncovers the ineffectiveness of naive applications of existing binarization algorithms and highlights the imperative role of salient weights in achieving low-bit quantization. Thus, PB-LLM filters a small ratio of salient weights during binarization, allocating them to higher-bit storage, i.e., partially-binarization. PB-LLM is extended to recover the capacities of quantized LMMs, by analyzing from the perspective of post-training quantization (PTQ) and quantization-aware training (QAT). Under PTQ, combining the concepts from GPTQ, we reconstruct the binarized weight matrix guided by the Hessian matrix and successfully recover the reasoning capacity of PB-LLM in low-bit. Under QAT, we freeze the salient weights during training, explore the derivation of optimal scaling factors crucial for minimizing the quantization error, and propose a scaling mechanism based on this derived scaling strategy for residual binarized weights. Those explorations and the developed methodologies significantly contribute to rejuvenating the performance of low-bit quantized LLMs and present substantial advancements in the field of network binarization for LLMs.The code is available at https://github.com/hahnyuan/BinaryLLM.","url_abs":"https://arxiv.org/abs/2310.00034v2","url_pdf":"https://arxiv.org/pdf/2310.00034v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pb-llm-partially-binarized-large-language","repo_url":"https://github.com/hahnyuan/binaryllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"pb-llm-partially-binarized-large-language","repo_url":"https://github.com/hahnyuan/pb-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"binarization","task_name":"Binarization"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.00034","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.00034"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hahnyuan/pb-llm","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hahnyuan/binaryllm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a9e7f2cdf016b88b","entry":"find_layers","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"gptq_pb/modelutils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/gptq_pb/modelutils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9e7f2cdf016b88b"}},{"code_sha256_prefix":"fab444ffcb40c0e4","entry":"get_scheduler","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"qat/run_qat.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/qat/run_qat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fab444ffcb40c0e4"}},{"code_sha256_prefix":"5cc6ee4fff500d54","entry":"prepare_model_for_eval","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cc6ee4fff500d54"}},{"code_sha256_prefix":"9fa7ef614ee2fada","entry":"prepare_model_for_training","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fa7ef614ee2fada"}},{"code_sha256_prefix":"50ceff9d34d96d60","entry":"quantize","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"gptq_pb/high_quant.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/gptq_pb/high_quant.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50ceff9d34d96d60"}},{"code_sha256_prefix":"1cbf0df5b5d26b2a","entry":"get_english_quote","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cbf0df5b5d26b2a"}},{"code_sha256_prefix":"a7cbf4fc49c71e03","entry":"get_ptb","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"gptq_pb/datautils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/gptq_pb/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7cbf4fc49c71e03"}},{"code_sha256_prefix":"b1795af698ba5bc6","entry":"get_qat_dataset","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1795af698ba5bc6"}},{"code_sha256_prefix":"783b484325d75ee8","entry":"get_redpajama_train","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"783b484325d75ee8"}},{"code_sha256_prefix":"729f957048af4a3f","entry":"get_tokenizer","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"gptq_pb/datautils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/gptq_pb/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"729f957048af4a3f"}},{"code_sha256_prefix":"011909315dfdbc2e","entry":"get_wikitext2","repo":"hahnyuan/binaryllm","repo_kind":"official","path":"gptq_pb/datautils.py","file_url":"https://github.com/hahnyuan/binaryllm/blob/HEAD/gptq_pb/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"011909315dfdbc2e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}