{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kvquant-towards-10-million-context-length-llm","title":"KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization","arxiv_id":"2401.18079","date":"2024-01-31","proceeding":null,"authors":["Coleman Hooper","Sehoon Kim","Hiva Mohammadzadeh","Michael W. Mahoney","Yakun Sophia Shao","Kurt Keutzer","Amir Gholami"],"abstract":"LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference. Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision. Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods: (i) Per-Channel Key Quantization, where we adjust the dimension along which we quantize the Key activations to better match the distribution; (ii) Pre-RoPE Key Quantization, where we quantize Key activations before the rotary positional embedding to mitigate its impact on quantization; (iii) Non-Uniform KV Cache Quantization, where we derive per-layer sensitivity-weighted non-uniform datatypes that better represent the distributions; and (iv) Per-Vector Dense-and-Sparse Quantization, where we isolate outliers separately for each vector to minimize skews in quantization ranges. By applying our method to the LLaMA, Llama-2, Llama-3, and Mistral models, we achieve < 0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4, outperforming existing approaches. Our method enables serving LLaMA-7B with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system. We develop custom CUDA kernels for KVQuant, showing that we can achieve up to ~1.7x speedups, compared to baseline fp16 matrix-vector multiplications, for the LLaMA-7B model.","url_abs":"https://arxiv.org/abs/2401.18079v5","url_pdf":"https://arxiv.org/pdf/2401.18079v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kvquant-towards-10-million-context-length-llm","repo_url":"https://github.com/squeezeailab/kvquant","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"kvquant-towards-10-million-context-length-llm","repo_url":"https://github.com/ruikangliu/IntactKV","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.18079","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.18079"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/SqueezeAILab/KVQuant","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruikangliu/IntactKV","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/squeezeailab/kvquant","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"1fe7913f2b94b5f0","entry":"SimQuant","repo":"squeezeailab/kvquant","repo_kind":"official","path":"quant/kvquant/simquant_module_quantizer.py","file_url":"https://github.com/squeezeailab/kvquant/blob/HEAD/quant/kvquant/simquant_module_quantizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fe7913f2b94b5f0"}},{"code_sha256_prefix":"71eb7727feb23c97","entry":"find_layers","repo":"SqueezeAILab/KVQuant","repo_kind":"official","path":"deployment/kvquant/modelutils.py","file_url":"https://github.com/SqueezeAILab/KVQuant/blob/HEAD/deployment/kvquant/modelutils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71eb7727feb23c97"}},{"code_sha256_prefix":"3e91fb91f5019b88","entry":"round_to_nearest_pole_sim","repo":"squeezeailab/kvquant","repo_kind":"official","path":"quant/kvquant/simquant_module_quantizer.py","file_url":"https://github.com/squeezeailab/kvquant/blob/HEAD/quant/kvquant/simquant_module_quantizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e91fb91f5019b88"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}