{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pyramidkv-dynamic-kv-cache-compression-based","title":"PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling","arxiv_id":"2406.02069","date":"2024-06-04","proceeding":null,"authors":["Zefan Cai","Yichi Zhang","Bofei Gao","Yuliang Liu","Yucheng Li","Tianyu Liu","Keming Lu","Wayne Xiong","Yue Dong","Junjie Hu","Wen Xiao"],"abstract":"In this study, we investigate whether attention-based information flow inside large language models (LLMs) is aggregated through noticeable patterns for long context processing. Our observations reveal that LLMs aggregate information through Pyramidal Information Funneling where attention is scattering widely in lower layers, progressively consolidating within specific contexts, and ultimately focusing on critical tokens (a.k.a massive activation or attention sink) in higher layers. Motivated by these insights, we developed PyramidKV, a novel and effective KV cache compression method. This approach dynamically adjusts the KV cache size across different layers, allocating more cache in lower layers and less in higher ones, diverging from traditional methods that maintain a uniform KV cache size. Our experimental evaluations, utilizing the LongBench benchmark, show that PyramidKV matches the performance of models with a full KV cache while retaining only 12% of the KV cache, thus significantly reducing memory usage. In scenarios emphasizing memory efficiency, where only 0.7% of the KV cache is maintained, PyramidKV surpasses other KV cache compression techniques, achieving up to a 20.5 absolute accuracy improvement on TREC dataset. In the Needle-in-a-Haystack experiment, PyramidKV outperforms competing methods in maintaining long-context comprehension in LLMs; notably, retaining just 128 KV cache entries enables the LLAMA-3-70B model to achieve 100.0 Acc. performance.","url_abs":"https://arxiv.org/abs/2406.02069v4","url_pdf":"https://arxiv.org/pdf/2406.02069v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pyramidkv-dynamic-kv-cache-compression-based","repo_url":"https://github.com/zefan-cai/pyramidkv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"pyramidkv-dynamic-kv-cache-compression-based","repo_url":"https://github.com/answerdotai/cold-compress","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pyramidkv-dynamic-kv-cache-compression-based","repo_url":"https://github.com/princeton-pli/prulong","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.02069","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02069"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/answerdotai/cold-compress","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-pli/prulong","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zefan-cai/pyramidkv","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":3,"ran":10,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/pyramidkv_utils.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/pyramidkv_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"6856f796db256e0a","entry":"build_stop_token_ids","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/eval_utils.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6856f796db256e0a"}},{"code_sha256_prefix":"042f3baac334e2d7","entry":"compress_minicache_pair","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/minicache.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/minicache.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"042f3baac334e2d7"}},{"code_sha256_prefix":"472d821ef38be37d","entry":"create_window_attention_mask","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"472d821ef38be37d"}},{"code_sha256_prefix":"50463bb67a0214ae","entry":"get_headwise_projections","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/headinfer.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/headinfer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50463bb67a0214ae"}},{"code_sha256_prefix":"ad54d3a439b1ed31","entry":"headinfer_slot","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/headinfer.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/headinfer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad54d3a439b1ed31"}},{"code_sha256_prefix":"4fd25dca7e4d0ba3","entry":"is_empty_past_key_values","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/generation_state.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/generation_state.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fd25dca7e4d0ba3"}},{"code_sha256_prefix":"3597e5670a759378","entry":"key_pruner_query_driven","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/pyramidkv_utils.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/pyramidkv_utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3597e5670a759378"}},{"code_sha256_prefix":"6211e835b472f7ad","entry":"maybe_repeat_kv_before_cache","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/pyramidkv_utils.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/pyramidkv_utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6211e835b472f7ad"}},{"code_sha256_prefix":"dbe56944ebe0f117","entry":"minicache_slerp","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/minicache.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/minicache.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbe56944ebe0f117"}},{"code_sha256_prefix":"b453a87c0268a786","entry":"reduce_nacl_proxy_scores","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/nacl.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/nacl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b453a87c0268a786"}},{"code_sha256_prefix":"0753fefeac56ae68","entry":"select_minicache_retention_indices","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/minicache.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/minicache.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0753fefeac56ae68"}},{"code_sha256_prefix":"6aa8f85d4f403b46","entry":"select_nacl_tokens","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/nacl.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/nacl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6aa8f85d4f403b46"}},{"code_sha256_prefix":"d291b649166e280d","entry":"str2bool","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/eval_utils.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d291b649166e280d"}},{"code_sha256_prefix":"554279ac03f73d79","entry":"select_nacl_proxy_indices","repo":"zefan-cai/pyramidkv","repo_kind":"official","path":"pyramidkv/nacl.py","file_url":"https://github.com/zefan-cai/pyramidkv/blob/HEAD/pyramidkv/nacl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"554279ac03f73d79"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}