{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/qllm-accurate-and-efficient-low-bitwidth","title":"QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models","arxiv_id":"2310.08041","date":"2023-10-12","proceeding":null,"authors":["Jing Liu","Ruihao Gong","Xiuying Wei","Zhiwei Dong","Jianfei Cai","Bohan Zhuang"],"abstract":"Large Language Models (LLMs) excel in NLP, but their demands hinder their widespread deployment. While Quantization-Aware Training (QAT) offers a solution, its extensive training costs make Post-Training Quantization (PTQ) a more practical approach for LLMs. In existing studies, activation outliers in particular channels are identified as the bottleneck to PTQ accuracy. They propose to transform the magnitudes from activations to weights, which however offers limited alleviation or suffers from unstable gradients, resulting in a severe performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs. QLLM introduces an adaptive channel reassembly technique that reallocates the magnitude of outliers to other channels, thereby mitigating their impact on the quantization range. This is achieved by channel disassembly and channel assembly, which first breaks down the outlier channels into several sub-channels to ensure a more balanced distribution of activation magnitudes. Then similar channels are merged to maintain the original channel number for efficiency. Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly. To further compensate for the performance loss caused by quantization, we propose an efficient tuning method that only learns a small number of low-rank weights while freezing the pre-trained quantized model. After training, these low-rank parameters can be fused into the frozen weights without affecting inference. Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM can obtain accurate quantized models efficiently. For example, QLLM quantizes the 4-bit LLaMA-2-70B within 10 hours on a single A100-80G GPU, outperforming the previous state-of-the-art method by 7.89% on the average accuracy across five zero-shot tasks.","url_abs":"https://arxiv.org/abs/2310.08041v3","url_pdf":"https://arxiv.org/pdf/2310.08041v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"qllm-accurate-and-efficient-low-bitwidth","repo_url":"https://github.com/modeltc/qllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"qllm-accurate-and-efficient-low-bitwidth","repo_url":"https://github.com/ziplab/qllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.08041","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08041"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/modeltc/qllm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziplab/qllm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":15,"ran":7,"repositories":1},"community":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"de4a0bd85bb546f8","entry":"assembly","repo":"modeltc/qllm","repo_kind":"official","path":"assembly/ca_module.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/assembly/ca_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de4a0bd85bb546f8"}},{"code_sha256_prefix":"356d0dfdff3db93a","entry":"do_nothing","repo":"modeltc/qllm","repo_kind":"official","path":"assembly/ca_module.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/assembly/ca_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"356d0dfdff3db93a"}},{"code_sha256_prefix":"904cd61df2fe8fb9","entry":"find_layers","repo":"modeltc/qllm","repo_kind":"official","path":"models/models_utils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/models/models_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"904cd61df2fe8fb9"}},{"code_sha256_prefix":"4df9168e0a0c38eb","entry":"make_disjoint_window","repo":"modeltc/qllm","repo_kind":"official","path":"models/models_utils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/models/models_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4df9168e0a0c38eb"}},{"code_sha256_prefix":"e5e341cb2fd0757b","entry":"sort_layers_by_params","repo":"modeltc/qllm","repo_kind":"official","path":"parallel_utils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/parallel_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5e341cb2fd0757b"}},{"code_sha256_prefix":"94587954e392b665","entry":"truncate_number","repo":"modeltc/qllm","repo_kind":"official","path":"models/transformation.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/models/transformation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94587954e392b665"}},{"code_sha256_prefix":"4e53fbbdfa160f7d","entry":"truncate_number","repo":"modeltc/qllm","repo_kind":"official","path":"models/models_utils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/models/models_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e53fbbdfa160f7d"}},{"code_sha256_prefix":"08e5e88401071307","entry":"apply_rotary_pos_emb_k","repo":"modeltc/qllm","repo_kind":"official","path":"models/int_llama_layer.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/models/int_llama_layer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08e5e88401071307"}},{"code_sha256_prefix":"69b6e7d6c2228088","entry":"apply_rotary_pos_emb_q","repo":"modeltc/qllm","repo_kind":"official","path":"models/int_llama_layer.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/models/int_llama_layer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69b6e7d6c2228088"}},{"code_sha256_prefix":"b4401e4eaf52f123","entry":"assign_layers_to_gpus","repo":"modeltc/qllm","repo_kind":"official","path":"parallel_utils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/parallel_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4401e4eaf52f123"}},{"code_sha256_prefix":"aa4f281a4d0a2c47","entry":"bipartite_soft_matching_x_w","repo":"modeltc/qllm","repo_kind":"official","path":"assembly/ca_module.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/assembly/ca_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa4f281a4d0a2c47"}},{"code_sha256_prefix":"a4eda67b643f2d95","entry":"gen_rand_orthos","repo":"wejoncy/QLLM","repo_kind":"community","path":"qllm/quantization/method.py","file_url":"https://github.com/wejoncy/QLLM/blob/HEAD/qllm/quantization/method.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4eda67b643f2d95"}},{"code_sha256_prefix":"90df7d0b5fc6bf9a","entry":"get_lowest_occupied_gpu","repo":"modeltc/qllm","repo_kind":"official","path":"parallel_utils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/parallel_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90df7d0b5fc6bf9a"}},{"code_sha256_prefix":"2b6e5673cf2914c8","entry":"get_pile","repo":"modeltc/qllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b6e5673cf2914c8"}},{"code_sha256_prefix":"1b68bae7cd1dd862","entry":"get_ptb","repo":"modeltc/qllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b68bae7cd1dd862"}},{"code_sha256_prefix":"78a5251d494aa3a6","entry":"get_wikitext2","repo":"modeltc/qllm","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/modeltc/qllm/blob/HEAD/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"78a5251d494aa3a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}