{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gear-an-efficient-kv-cache-compression","title":"GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM","arxiv_id":"2403.05527","date":"2024-03-08","proceeding":null,"authors":["Hao Kang","Qingru Zhang","Souvik Kundu","Geonhwa Jeong","Zaoxing Liu","Tushar Krishna","Tuo Zhao"],"abstract":"Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of each step, resulting in critical deviation in model generation and deterioration of performance. To tackle this challenge, we propose GEAR, an efficient KV cache compression framework that achieves near-lossless high-ratio compression. GEAR first applies quantization to majority of entries of similar magnitudes to ultra-low precision. It then employs a low rank matrix to approximate the quantization error, and a sparse matrix to remedy individual errors from outlier entries. By adeptly integrating three techniques, GEAR is able to fully exploit their synergistic potentials. Our experiments demonstrate that compared to alternatives, GEAR achieves near-lossless 4-bit KV cache compression with up to 2.38x throughput improvement, while reducing peak-memory size up to 2.29x. Our code is publicly available at https://github.com/HaoKang-Timmy/GEAR.","url_abs":"https://arxiv.org/abs/2403.05527v4","url_pdf":"https://arxiv.org/pdf/2403.05527v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gear-an-efficient-kv-cache-compression","repo_url":"https://github.com/haokang-timmy/gear","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gear-an-efficient-kv-cache-compression","repo_url":"https://github.com/opengear-project/gear","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.05527","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05527"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haokang-timmy/gear","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengear-project/gear","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran":13,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":19,"ran":13,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"opengear-project/gear","repo_kind":"listed","path":"GenerationBench/GenerationTest/GEARLM/Simulated/modeling_llama_ablation.py","file_url":"https://github.com/opengear-project/gear/blob/HEAD/GenerationBench/GenerationTest/GEARLM/Simulated/modeling_llama_ablation.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"opengear-project/gear","repo_kind":"listed","path":"GenerationBench/GenerationTest/GEARLM/Simulated/modeling_llama_ablation.py","file_url":"https://github.com/opengear-project/gear/blob/HEAD/GenerationBench/GenerationTest/GEARLM/Simulated/modeling_llama_ablation.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"fd8719af3db2c5ae","entry":"calculate_zeros_width","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/qmodule.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/qmodule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd8719af3db2c5ae"}},{"code_sha256_prefix":"a9baefdb784da628","entry":"dequant_weight","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/gemv.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/gemv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9baefdb784da628"}},{"code_sha256_prefix":"4d4c20e8d1e678ef","entry":"dequant_weight_outer","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/gemv.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/gemv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d4c20e8d1e678ef"}},{"code_sha256_prefix":"3bbfc4445eea20a9","entry":"extract_ans","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_bbh_cot.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_bbh_cot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bbfc4445eea20a9"}},{"code_sha256_prefix":"fdbcb8662fd149ba","entry":"extract_ans_mmlu","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_aqua_cot.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_aqua_cot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdbcb8662fd149ba"}},{"code_sha256_prefix":"b96dbfc20b18beb7","entry":"extract_bbh_ans","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_aqua_cot.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_aqua_cot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b96dbfc20b18beb7"}},{"code_sha256_prefix":"3e8b5053d14e5887","entry":"if_not_then_None","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/modeling_llamagear.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/modeling_llamagear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e8b5053d14e5887"}},{"code_sha256_prefix":"ad85b288e9acb9c2","entry":"make_divisible","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/qmodule.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/qmodule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad85b288e9acb9c2"}},{"code_sha256_prefix":"901a3668958c9deb","entry":"quant_and_pack_kcache","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/new_pack.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/new_pack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"901a3668958c9deb"}},{"code_sha256_prefix":"95a9c97f6ea7d050","entry":"quant_and_pack_vcache","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/new_pack.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/new_pack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95a9c97f6ea7d050"}},{"code_sha256_prefix":"c6b9d9348efc20dc","entry":"reindent","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/timeit_v2.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/timeit_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6b9d9348efc20dc"}},{"code_sha256_prefix":"b9bb89598a35cb87","entry":"repeat","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/timeit_v2.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/timeit_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9bb89598a35cb87"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"opengear-project/gear","repo_kind":"listed","path":"GenerationBench/GenerationTest/GEARLM/Simulated/modeling_llama_ablation.py","file_url":"https://github.com/opengear-project/gear/blob/HEAD/GenerationBench/GenerationTest/GEARLM/Simulated/modeling_llama_ablation.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"c119c70eed02c785","entry":"test_answer_mmlu_","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_bbh_cot.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_bbh_cot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c119c70eed02c785"}},{"code_sha256_prefix":"b6a8148f2f12adc9","entry":"evaluate_pred_answer","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_gsm8k.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_gsm8k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6a8148f2f12adc9"}},{"code_sha256_prefix":"eb8e4a7084fe84dc","entry":"load_model_tokenizer","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_aqua_cot.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_aqua_cot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb8e4a7084fe84dc"}},{"code_sha256_prefix":"d93eb10d932ff2c8","entry":"parse_pred_ans","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_gsm8k.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_gsm8k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d93eb10d932ff2c8"}},{"code_sha256_prefix":"b6065f749757a201","entry":"test_answer","repo":"haokang-timmy/gear","repo_kind":"official","path":"GenerationBench/GenerationTest/evaluation_gsm8k.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/GenerationBench/GenerationTest/evaluation_gsm8k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6065f749757a201"}},{"code_sha256_prefix":"c743c02d9c56ac78","entry":"timeit","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/timeit_v2.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/timeit_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c743c02d9c56ac78"}},{"code_sha256_prefix":"e493c68ae46ac587","entry":"unpack_and_dequant_kcache","repo":"haokang-timmy/gear","repo_kind":"official","path":"cuda_supported_gear/quant/new_pack.py","file_url":"https://github.com/haokang-timmy/gear/blob/HEAD/cuda_supported_gear/quant/new_pack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e493c68ae46ac587"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}