{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omniquant-omnidirectionally-calibrated","title":"OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models","arxiv_id":"2308.13137","date":"2023-08-25","proceeding":null,"authors":["Wenqi Shao","Mengzhao Chen","Zhaoyang Zhang","Peng Xu","Lirui Zhao","Zhiqian Li","Kaipeng Zhang","Peng Gao","Yu Qiao","Ping Luo"],"abstract":"Large language models (LLMs) have revolutionized natural language processing tasks. However, their practical deployment is hindered by their immense memory and computation requirements. Although recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM, they hand-craft quantization parameters, leading to low performance, especially in extremely low-bit quantization. To tackle this issue, we introduce an Omnidirectionally calibrated Quantization (\\textbf{OmniQuant}) technique for LLMs, which achieves good performance in diverse quantization settings while maintaining the computational efficiency of PTQ by efficiently optimizing various quantization parameters. OmniQuant comprises two innovative components including Learnable Weight Clipping (LWC) and Learnable Equivalent Transformation (LET). LWC modulates the extreme values of weights by optimizing the clipping threshold. Meanwhile, LET tackles activation outliers by shifting the challenge of quantization from activations to weights. Operating within a differentiable framework using block-wise error minimization, OmniQuant can optimize the quantization process efficiently for both weight-only and weight-activation quantization. For instance, the LLaMA-2 model family size 7-70B can be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using 128 samples. Extensive experiments validate OmniQuant's superior performance across diverse quantization configurations such as W4A4 (4-bit weight, 4-bit activation), W6A6, W4A16, W3A16, and W2A16. Additionally, OmniQuant demonstrates effectiveness in instruction-tuned models and delivers notable improvements in inference speed and memory reduction on real devices. Codes are available at \\url{https://github.com/OpenGVLab/OmniQuant}.","url_abs":"https://arxiv.org/abs/2308.13137v3","url_pdf":"https://arxiv.org/pdf/2308.13137v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omniquant-omnidirectionally-calibrated","repo_url":"https://github.com/opengvlab/omniquant","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"omniquant-omnidirectionally-calibrated","repo_url":"https://github.com/greenbitai/low_bit_llama","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.13137","atlas_url":"https://app.syntology.ai/?focus=2308.13137","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.13137"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OpenGVLab/OmniQuant","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/omniquant","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/greenbitai/low_bit_llama","reach":null}],"summary":{"ran":7,"ran_draft_wrong":2,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d4c7f5b91441ced1","entry":"QuantLinear","repo":"greenbitai/low_bit_llama","repo_kind":"listed","path":"model.py","file_url":"https://github.com/greenbitai/low_bit_llama/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d4c7f5b91441ced1"}},{"code_sha256_prefix":"c07dba9529e937b7","entry":"ampscaler_get_grad_norm","repo":"opengvlab/omniquant","repo_kind":"official","path":"utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c07dba9529e937b7"}},{"code_sha256_prefix":"904cd61df2fe8fb9","entry":"find_layers","repo":"opengvlab/omniquant","repo_kind":"official","path":"models/models_utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/models/models_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"904cd61df2fe8fb9"}},{"code_sha256_prefix":"bd1c7aaaa05aeaf3","entry":"get_act_scales","repo":"opengvlab/omniquant","repo_kind":"official","path":"generate_act_scale_shift.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/generate_act_scale_shift.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd1c7aaaa05aeaf3"}},{"code_sha256_prefix":"550d556e74fb20f7","entry":"get_act_shifts","repo":"opengvlab/omniquant","repo_kind":"official","path":"generate_act_scale_shift.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/generate_act_scale_shift.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"550d556e74fb20f7"}},{"code_sha256_prefix":"4df9168e0a0c38eb","entry":"make_disjoint_window","repo":"opengvlab/omniquant","repo_kind":"official","path":"models/models_utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/models/models_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4df9168e0a0c38eb"}},{"code_sha256_prefix":"aaa7723c864e0a5e","entry":"sort_layers_by_params","repo":"opengvlab/omniquant","repo_kind":"official","path":"parallel_utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/parallel_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aaa7723c864e0a5e"}},{"code_sha256_prefix":"94587954e392b665","entry":"truncate_number","repo":"opengvlab/omniquant","repo_kind":"official","path":"models/transformation.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/models/transformation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94587954e392b665"}},{"code_sha256_prefix":"4e53fbbdfa160f7d","entry":"truncate_number","repo":"opengvlab/omniquant","repo_kind":"official","path":"models/models_utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/models/models_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e53fbbdfa160f7d"}},{"code_sha256_prefix":"7e3ef1ed347af0f3","entry":"assign_layers_to_gpus","repo":"opengvlab/omniquant","repo_kind":"official","path":"parallel_utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/parallel_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e3ef1ed347af0f3"}},{"code_sha256_prefix":"a041d7c9b413c995","entry":"build_model_and_tokenizer","repo":"opengvlab/omniquant","repo_kind":"official","path":"generate_act_scale_shift.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/generate_act_scale_shift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a041d7c9b413c995"}},{"code_sha256_prefix":"87877d0c60858bff","entry":"create_logger","repo":"opengvlab/omniquant","repo_kind":"official","path":"utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87877d0c60858bff"}},{"code_sha256_prefix":"3c6439a3c61dd2bb","entry":"get_lowest_occupied_gpu","repo":"opengvlab/omniquant","repo_kind":"official","path":"parallel_utils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/parallel_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c6439a3c61dd2bb"}},{"code_sha256_prefix":"3b51341ed1a0b73e","entry":"get_pile","repo":"opengvlab/omniquant","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b51341ed1a0b73e"}},{"code_sha256_prefix":"bd4f416e271c2fb5","entry":"get_ptb","repo":"opengvlab/omniquant","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd4f416e271c2fb5"}},{"code_sha256_prefix":"406294899f965c2f","entry":"get_wikitext2","repo":"opengvlab/omniquant","repo_kind":"official","path":"datautils.py","file_url":"https://github.com/opengvlab/omniquant/blob/HEAD/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"406294899f965c2f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}