{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimizing-kv-cache-eviction-in-llms-adaptive","title":"Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference","arxiv_id":"2407.11550","date":"2024-07-16","proceeding":null,"authors":["Yuan Feng","Junlin Lv","Yukun Cao","Xike Xie","S. Kevin Zhou"],"abstract":"Large Language Models have excelled in various domains but face efficiency challenges due to the growing Key-Value (KV) cache required for long-sequence inference. Recent efforts aim to reduce KV cache size by evicting vast non-critical cache elements during runtime while preserving generation quality. However, these methods typically allocate compression budgets uniformly across all attention heads, ignoring the unique attention patterns of each head. In this paper, we establish a theoretical loss upper bound between pre- and post-eviction attention output, explaining the optimization target of prior cache eviction methods, while guiding the optimization of adaptive budget allocation. Base on this, we propose {\\it Ada-KV}, the first head-wise adaptive budget allocation strategy. It offers plug-and-play benefits, enabling seamless integration with prior cache eviction methods. Extensive evaluations on 13 datasets from Ruler and 16 datasets from LongBench, all conducted under both question-aware and question-agnostic scenarios, demonstrate substantial quality improvements over existing methods.","url_abs":"https://arxiv.org/abs/2407.11550v4","url_pdf":"https://arxiv.org/pdf/2407.11550v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimizing-kv-cache-eviction-in-llms-adaptive","repo_url":"https://github.com/ffy0/adakv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"optimizing-kv-cache-eviction-in-llms-adaptive","repo_url":"https://github.com/NVIDIA/kvpress","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.11550","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.11550"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/FFY0/AdaKV","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ffy0/adakv","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVIDIA/kvpress","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"2cc48d857fba9681","entry":"config_compress","repo":"FFY0/AdaKV","repo_kind":"official","path":"adaptive_snapkv/monkeypatch/monkeypatch.py","file_url":"https://github.com/FFY0/AdaKV/blob/HEAD/adaptive_snapkv/monkeypatch/monkeypatch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2cc48d857fba9681"}},{"code_sha256_prefix":"4fae14db255bbbe6","entry":"AdaKVPress","repo":"NVIDIA/kvpress","repo_kind":"official","path":"kvpress/presses/adakv_press.py","file_url":"https://github.com/NVIDIA/kvpress/blob/HEAD/kvpress/presses/adakv_press.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4fae14db255bbbe6"}},{"code_sha256_prefix":"016e62e773c90c11","entry":"AdaptiveSnapKVCluster","repo":"ffy0/adakv","repo_kind":"official","path":"adaptive_snapkv/monkeypatch/snapkv_utils.py","file_url":"https://github.com/ffy0/adakv/blob/HEAD/adaptive_snapkv/monkeypatch/snapkv_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"016e62e773c90c11"}},{"code_sha256_prefix":"c4f6af4dd0288d5c","entry":"BasePress","repo":"NVIDIA/kvpress","repo_kind":"official","path":"kvpress/presses/adakv_press.py","file_url":"https://github.com/NVIDIA/kvpress/blob/HEAD/kvpress/presses/adakv_press.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c4f6af4dd0288d5c"}},{"code_sha256_prefix":"1fa519d26ac61a5b","entry":"ScorerPress","repo":"NVIDIA/kvpress","repo_kind":"official","path":"kvpress/presses/adakv_press.py","file_url":"https://github.com/NVIDIA/kvpress/blob/HEAD/kvpress/presses/adakv_press.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fa519d26ac61a5b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}