{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zipcache-accurate-and-efficient-kv-cache","title":"ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification","arxiv_id":"2405.14256","date":"2024-05-23","proceeding":null,"authors":["Yefei He","Luoming Zhang","Weijia Wu","Jing Liu","Hong Zhou","Bohan Zhuang"],"abstract":"KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance. However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. In this paper, we present ZipCache, an accurate and efficient KV cache quantization method for LLMs. First, we construct a strong baseline for quantizing KV cache. Through the proposed channel-separable tokenwise quantization scheme, the memory overhead of quantization parameters are substantially reduced compared to fine-grained groupwise quantization. To enhance the compression ratio, we propose normalized attention score as an effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention. Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and minimal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by $4.98\\times$, with only a $0.38\\%$ drop in accuracy. In terms of efficiency, ZipCache also showcases a $37.3\\%$ reduction in prefill-phase latency, a $56.9\\%$ reduction in decoding-phase latency, and a $19.8\\%$ reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of $4096$.","url_abs":"https://arxiv.org/abs/2405.14256v1","url_pdf":"https://arxiv.org/pdf/2405.14256v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zipcache-accurate-and-efficient-kv-cache","repo_url":"https://github.com/thisisbillhe/zipcache","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.14256","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14256"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thisisbillhe/zipcache","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"listed":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"thisisbillhe/zipcache","repo_kind":"listed","path":"zipcache/models/modeling_llama.py","file_url":"https://github.com/thisisbillhe/zipcache/blob/HEAD/zipcache/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"thisisbillhe/zipcache","repo_kind":"listed","path":"zipcache/models/modeling_llama.py","file_url":"https://github.com/thisisbillhe/zipcache/blob/HEAD/zipcache/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"thisisbillhe/zipcache","repo_kind":"listed","path":"zipcache/models/modeling_llama.py","file_url":"https://github.com/thisisbillhe/zipcache/blob/HEAD/zipcache/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"b167b70daca2e13d","entry":"transfer_2bit_to_8bit_batchwise","repo":"thisisbillhe/zipcache","repo_kind":"listed","path":"zipcache/models/CompressUtils/compress_function.py","file_url":"https://github.com/thisisbillhe/zipcache/blob/HEAD/zipcache/models/CompressUtils/compress_function.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b167b70daca2e13d"}},{"code_sha256_prefix":"8e36b8276cd96cd4","entry":"transfer_8bit_to_2bit_batchwise","repo":"thisisbillhe/zipcache","repo_kind":"listed","path":"zipcache/models/CompressUtils/compress_function.py","file_url":"https://github.com/thisisbillhe/zipcache/blob/HEAD/zipcache/models/CompressUtils/compress_function.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e36b8276cd96cd4"}},{"code_sha256_prefix":"3b8503ce511cc73c","entry":"transfer_8bit_to_4bit","repo":"thisisbillhe/zipcache","repo_kind":"listed","path":"zipcache/models/CompressUtils/compress_function.py","file_url":"https://github.com/thisisbillhe/zipcache/blob/HEAD/zipcache/models/CompressUtils/compress_function.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b8503ce511cc73c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}