{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/not-all-heads-matter-a-head-level-kv-cache","title":"Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning","arxiv_id":"2410.19258","date":"2024-10-25","proceeding":null,"authors":["Yu Fu","Zefan Cai","Abedelkadir Asi","Wayne Xiong","Yue Dong","Wen Xiao"],"abstract":"Key-Value (KV) caching is a common technique to enhance the computational efficiency of Large Language Models (LLMs), but its memory overhead grows rapidly with input length. Prior work has shown that not all tokens are equally important for text generation, proposing layer-level KV cache compression to selectively retain key information. Recognizing the distinct roles of attention heads in generation, we propose HeadKV, a head-level KV cache compression method, and HeadKV-R2, which leverages a novel contextual reasoning ability estimation for compression. Our approach operates at the level of individual heads, estimating their importance for contextual QA tasks that require both retrieval and reasoning capabilities. Extensive experiments across diverse benchmarks (LongBench, LooGLE), model architectures (e.g., Llama-3-8B-Instruct, Mistral-7B-Instruct), and long-context abilities tests demonstrate that our head-level KV cache compression significantly outperforms strong baselines, particularly in low-resource settings (KV size = 64 & 128). Notably, our method retains just 1.5% of the KV cache while achieving 97% of the performance of the full KV cache on the contextual question answering benchmark.Codes are available at https://github.com/FYYFU/HeadKV","url_abs":"https://arxiv.org/abs/2410.19258v3","url_pdf":"https://arxiv.org/pdf/2410.19258v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"not-all-heads-matter-a-head-level-kv-cache","repo_url":"https://github.com/fyyfu/headkv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.19258","atlas_url":"https://app.syntology.ai/?focus=2410.19258","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.19258"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fyyfu/headkv","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"fyyfu/headkv","repo_kind":"official","path":"headkv/snapkv_utils.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/headkv/snapkv_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"3ddd62fbad41e698","entry":"build_chat","repo":"fyyfu/headkv","repo_kind":"official","path":"run_longbench.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/run_longbench.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ddd62fbad41e698"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"fyyfu/headkv","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"fyyfu/headkv","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"fyyfu/headkv","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"1ff384226516c0cb","entry":"fixed_LlamaModel_forward","repo":"fyyfu/headkv","repo_kind":"official","path":"headkv/fixed_llama_hijack.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/headkv/fixed_llama_hijack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ff384226516c0cb"}},{"code_sha256_prefix":"ea5369d01a779749","entry":"fixed_MistralModel_forward","repo":"fyyfu/headkv","repo_kind":"official","path":"headkv/fixed_mistral_hijack.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/headkv/fixed_mistral_hijack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea5369d01a779749"}},{"code_sha256_prefix":"f678f94474df1f5a","entry":"get_cuda_bare_metal_version","repo":"fyyfu/headkv","repo_kind":"official","path":"csrc/build.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/csrc/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f678f94474df1f5a"}},{"code_sha256_prefix":"db4b28c72d99c354","entry":"parse_args","repo":"fyyfu/headkv","repo_kind":"official","path":"eval_babilong.py","file_url":"https://github.com/fyyfu/headkv/blob/HEAD/eval_babilong.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db4b28c72d99c354"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}