{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unlocking-data-free-low-bit-quantization-with","title":"Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression","arxiv_id":"2405.12591","date":"2024-05-21","proceeding":null,"authors":["Peiyu Liu","Ze-Feng Gao","Wayne Xin Zhao","Yipeng Ma","Tao Wang","Ji-Rong Wen"],"abstract":"Key-value~(KV) caching is an important technique to accelerate the inference of large language models~(LLMs), but incurs significant memory overhead. To compress the size of KV cache, existing methods often compromise precision or require extra data for calibration, limiting their practicality in LLM deployment. In this paper, we introduce \\textbf{DecoQuant}, a novel data-free low-bit quantization technique based on tensor decomposition methods, to effectively compress KV cache. Our core idea is to adjust the outlier distribution of the original matrix by performing tensor decomposition, so that the quantization difficulties are migrated from the matrix to decomposed local tensors. Specially, we find that outliers mainly concentrate on small local tensors, while large tensors tend to have a narrower value range. Based on this finding, we propose to apply low-bit quantization to the large tensor, while maintaining high-precision representation for the small tensor. Furthermore, we utilize the proposed quantization method to compress the KV cache of LLMs to accelerate the inference and develop an efficient dequantization kernel tailored specifically for DecoQuant. Through extensive experiments, DecoQuant demonstrates remarkable efficiency gains, showcasing up to a $\\sim$75\\% reduction in memory footprint while maintaining comparable generation quality.","url_abs":"https://arxiv.org/abs/2405.12591v1","url_pdf":"https://arxiv.org/pdf/2405.12591v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unlocking-data-free-low-bit-quantization-with","repo_url":"https://github.com/lpyhdzx/DecoQuant_code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"tensor-decomposition","task_name":"Tensor Decomposition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.12591","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.12591"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lpyhdzx/DecoQuant_code","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"55c898dd0bb4836e","entry":"get_last_checkpoint_or_last_model","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/utils.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55c898dd0bb4836e"}},{"code_sha256_prefix":"383d339873e8f63f","entry":"get_parameter_number","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/evaluate_icl_ori.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/evaluate_icl_ori.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"383d339873e8f63f"}},{"code_sha256_prefix":"231f5cf156c8b486","entry":"parse_checkpoint_step","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/utils.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"231f5cf156c8b486"}},{"code_sha256_prefix":"c1cb961b4f5cb75c","entry":"quantize_activation_per_token_absmax","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/models/fake_quant.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/models/fake_quant.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1cb961b4f5cb75c"}},{"code_sha256_prefix":"39396670c3373c8e","entry":"quantize_weight_per_channel_absmax","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/models/fake_quant.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/models/fake_quant.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39396670c3373c8e"}},{"code_sha256_prefix":"dd3493503cb95888","entry":"quantize_weight_per_tensor_absmax","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/models/fake_quant.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/models/fake_quant.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd3493503cb95888"}},{"code_sha256_prefix":"3fed3338f2ffb08f","entry":"get_act_scales","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"language_model/smoothquant/calibration.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/language_model/smoothquant/calibration.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3fed3338f2ffb08f"}},{"code_sha256_prefix":"062aa2adf5c6b989","entry":"get_dataset","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/icl_dataset_loading.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/icl_dataset_loading.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"062aa2adf5c6b989"}},{"code_sha256_prefix":"46b0ba2dfd30560a","entry":"get_model_tokenizer_device_isac","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"icl/evaluate_icl_ori.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/icl/evaluate_icl_ori.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46b0ba2dfd30560a"}},{"code_sha256_prefix":"a5a75a01c343fb3b","entry":"get_static_decoder_layer_scales","repo":"lpyhdzx/DecoQuant_code","repo_kind":"official","path":"language_model/smoothquant/calibration.py","file_url":"https://github.com/lpyhdzx/DecoQuant_code/blob/HEAD/language_model/smoothquant/calibration.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5a75a01c343fb3b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}