{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simlayerkv-a-simple-framework-for-layer-level","title":"SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction","arxiv_id":"2410.13846","date":"2024-10-17","proceeding":null,"authors":["Xuan Zhang","Cunxiao Du","Chao Du","Tianyu Pang","Wei Gao","Min Lin"],"abstract":"Recent advancements in large language models (LLMs) have extended their capabilities to handle long contexts. However, increasing the number of model layers and the length of input sequences significantly escalates the memory required to store key-value (KV) cache, posing challenges for efficient inference. To mitigate this issue, we present SimLayerKV, a simple yet effective method that reduces inter-layer KV cache redundancies by selectively dropping cache in identified lazy layers. Our approach is based on the observation that certain layers in long-context LLMs exhibit \"lazy\" behavior, contributing less to modeling long-range dependencies compared to non-lazy layers. By analyzing attention weight patterns, we find that the behavior of these lazy layers is consistent across tokens during generation for a given input. This insight motivates our SimLayerKV, which identifies lazy layers and reduces their KV cache accordingly. SimLayerKV is training-free, generalizable, and can be implemented with only seven lines of code. We conduct extensive experiments on three representative LLMs, e.g., LLaMA2-7B, LLaMA3-8B, and Mistral-7B across 16 tasks from the LongBench benchmark. The results demonstrate that SimLayerKV achieves a KV cache compression ratio of 5$\\times$ with only a 1.2% performance drop when combined with 4-bit quantization. Our code is available at https://github.com/sail-sg/SimLayerKV.","url_abs":"https://arxiv.org/abs/2410.13846v1","url_pdf":"https://arxiv.org/pdf/2410.13846v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simlayerkv-a-simple-framework-for-layer-level","repo_url":"https://github.com/sail-sg/simlayerkv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.13846","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.13846"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/simlayerkv","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_honours":1,"ran_draft_wrong":2,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"109fdfadb74e7b7e","entry":"apply_rotary_pos_emb_single","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/SimLayerKV_attention_llama.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/SimLayerKV_attention_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"109fdfadb74e7b7e"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/metrics.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"a5e6881df94a8792","entry":"parse_args","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/pred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5e6881df94a8792"}},{"code_sha256_prefix":"4489113b536ca6eb","entry":"post_process","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/pred.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4489113b536ca6eb"}},{"code_sha256_prefix":"fa0ead5eb565a101","entry":"relative_top_filter","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/SimLayerKV_llama.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/SimLayerKV_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa0ead5eb565a101"}},{"code_sha256_prefix":"fa013be990ccedab","entry":"repeat_kv_quant","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/KIVI/models/mistral_kivi.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/KIVI/models/mistral_kivi.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa013be990ccedab"}},{"code_sha256_prefix":"15435bf612fd7f6f","entry":"scaled_dot_product_attention","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/SimLayerKV_attention_llama.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/SimLayerKV_attention_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15435bf612fd7f6f"}},{"code_sha256_prefix":"996f634b37f61510","entry":"scaled_dot_product_attention","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/SimLayerKV_attention_qwen.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/SimLayerKV_attention_qwen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"996f634b37f61510"}},{"code_sha256_prefix":"3a5da3106b298ea0","entry":"slerp","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/SimLayerKV_llama.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/SimLayerKV_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a5da3106b298ea0"}},{"code_sha256_prefix":"324506640e971548","entry":"build_chat","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/pred.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/pred.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"324506640e971548"}},{"code_sha256_prefix":"7f2e8a952b2a3486","entry":"ori_forward","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/SimLayerKV_attention_llama.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/SimLayerKV_attention_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f2e8a952b2a3486"}},{"code_sha256_prefix":"8bef8b8d37f07b4e","entry":"ori_forward","repo":"sail-sg/simlayerkv","repo_kind":"official","path":"LongBench/SimLayerKV_attention_qwen.py","file_url":"https://github.com/sail-sg/simlayerkv/blob/HEAD/LongBench/SimLayerKV_attention_qwen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bef8b8d37f07b4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}