{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/qjl-1-bit-quantized-jl-transform-for-kv-cache","title":"QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead","arxiv_id":"2406.03482","date":"2024-06-05","proceeding":null,"authors":["Amir Zandieh","Majid Daliri","Insu Han"],"abstract":"Serving LLMs requires substantial memory due to the storage requirements of Key-Value (KV) embeddings in the KV cache, which grows with sequence length. An effective approach to compress KV cache is quantization. However, traditional quantization methods face significant memory overhead due to the need to store quantization constants (at least a zero point and a scale) in full precision per data block. Depending on the block size, this overhead can add 1 or 2 bits per quantized number. We introduce QJL, a new quantization approach that consists of a Johnson-Lindenstrauss (JL) transform followed by sign-bit quantization. In contrast to existing methods, QJL eliminates memory overheads by removing the need for storing quantization constants. We propose an asymmetric estimator for the inner product of two vectors and demonstrate that applying QJL to one vector and a standard JL transform without quantization to the other provides an unbiased estimator with minimal distortion. We have developed an efficient implementation of the QJL sketch and its corresponding inner product estimator, incorporating a lightweight CUDA kernel for optimized computation. When applied across various LLMs and NLP tasks to quantize the KV cache to only 3 bits, QJL demonstrates a more than fivefold reduction in KV cache memory usage without compromising accuracy, all while achieving faster runtime. Codes are available at \\url{https://github.com/amirzandieh/QJL}.","url_abs":"https://arxiv.org/abs/2406.03482v2","url_pdf":"https://arxiv.org/pdf/2406.03482v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"qjl-1-bit-quantized-jl-transform-for-kv-cache","repo_url":"https://github.com/amirzandieh/qjl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.03482","atlas_url":"https://app.syntology.ai/?focus=2406.03482","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.03482"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amirzandieh/qjl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"amirzandieh/qjl","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/amirzandieh/qjl/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"amirzandieh/qjl","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/amirzandieh/qjl/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"amirzandieh/qjl","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/amirzandieh/qjl/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"ca066a4c088c8cf5","entry":"parse_args","repo":"amirzandieh/qjl","repo_kind":"official","path":"eval_long_bench.py","file_url":"https://github.com/amirzandieh/qjl/blob/HEAD/eval_long_bench.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca066a4c088c8cf5"}},{"code_sha256_prefix":"19035055b71c9787","entry":"repeat_kv_quant","repo":"amirzandieh/qjl","repo_kind":"official","path":"models/llama3_utils_qjl.py","file_url":"https://github.com/amirzandieh/qjl/blob/HEAD/models/llama3_utils_qjl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19035055b71c9787"}},{"code_sha256_prefix":"8bf52d5d7d55d38f","entry":"greedy_generate","repo":"amirzandieh/qjl","repo_kind":"official","path":"plot_memory_peak.py","file_url":"https://github.com/amirzandieh/qjl/blob/HEAD/plot_memory_peak.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8bf52d5d7d55d38f"}},{"code_sha256_prefix":"c541ae13a407a3ce","entry":"greedy_generate","repo":"amirzandieh/qjl","repo_kind":"official","path":"plot_runtime.py","file_url":"https://github.com/amirzandieh/qjl/blob/HEAD/plot_runtime.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c541ae13a407a3ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}