{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-efficacy-of-eviction-policy-for-key","title":"On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference","arxiv_id":"2402.06262","date":"2024-02-09","proceeding":null,"authors":["Siyu Ren","Kenny Q. Zhu"],"abstract":"Despite the recent success associated with Large Language Models (LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive memory and computational demands. In addition to model parameters, the key-value cache is also stored in GPU memory, growing linearly with batch size and sequence length. As a remedy, recent works have proposed various eviction policies for maintaining the overhead of key-value cache under a given budget. This paper embarks on the efficacy of existing eviction policies in terms of importance score calculation and eviction scope construction. We identify the deficiency of prior policies in these two aspects and introduce RoCo, a robust cache omission policy based on temporal attention scores and robustness measures. Extensive experimentation spanning prefilling and auto-regressive decoding stages validates the superiority of RoCo. Finally, we release EasyKV, a versatile software package dedicated to user-friendly key-value constrained generative inference. Code available at https://github.com/DRSY/EasyKV.","url_abs":"https://arxiv.org/abs/2402.06262v2","url_pdf":"https://arxiv.org/pdf/2402.06262v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-efficacy-of-eviction-policy-for-key","repo_url":"https://github.com/drsy/easykv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.06262","atlas_url":"https://app.syntology.ai/?focus=2402.06262","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.06262"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/drsy/easykv","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran":5,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"drsy/easykv","repo_kind":"official","path":"easykv/llama_patch.py","file_url":"https://github.com/drsy/easykv/blob/HEAD/easykv/llama_patch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"79567d2173f30f8d","entry":"apply_grouped_rotary_pos_emb","repo":"drsy/easykv","repo_kind":"official","path":"easykv/llama_patch.py","file_url":"https://github.com/drsy/easykv/blob/HEAD/easykv/llama_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79567d2173f30f8d"}},{"code_sha256_prefix":"8869f3ae73d37f2c","entry":"apply_rotary_pos_emb_sep","repo":"drsy/easykv","repo_kind":"official","path":"easykv/mistral_patch.py","file_url":"https://github.com/drsy/easykv/blob/HEAD/easykv/mistral_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8869f3ae73d37f2c"}},{"code_sha256_prefix":"afa07c0f7e539ce7","entry":"cache_size","repo":"drsy/easykv","repo_kind":"official","path":"easykv/easykv.py","file_url":"https://github.com/drsy/easykv/blob/HEAD/easykv/easykv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afa07c0f7e539ce7"}},{"code_sha256_prefix":"5f9c91cf82e9ea48","entry":"entropy","repo":"drsy/easykv","repo_kind":"official","path":"easykv/easykv.py","file_url":"https://github.com/drsy/easykv/blob/HEAD/easykv/easykv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f9c91cf82e9ea48"}},{"code_sha256_prefix":"6264652451270da5","entry":"relu_normalize","repo":"drsy/easykv","repo_kind":"official","path":"easykv/easykv.py","file_url":"https://github.com/drsy/easykv/blob/HEAD/easykv/easykv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6264652451270da5"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"drsy/easykv","repo_kind":"official","path":"easykv/llama_patch.py","file_url":"https://github.com/drsy/easykv/blob/HEAD/easykv/llama_patch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}