{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/snapkv-llm-knows-what-you-are-looking-for","title":"SnapKV: LLM Knows What You are Looking for Before Generation","arxiv_id":"2404.14469","date":"2024-04-22","proceeding":null,"authors":["Yuhong Li","Yingbing Huang","Bowen Yang","Bharat Venkitesh","Acyr Locatelli","Hanchen Ye","Tianle Cai","Patrick Lewis","Deming Chen"],"abstract":"Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance. However, the growth of the KV cache in response to increasing input length poses challenges to memory and time efficiency. To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-free approach that efficiently minimizes KV cache size while still delivering comparable performance in real-world applications. We discover that each attention head in the model consistently focuses on specific prompt attention features during generation. Meanwhile, this robust pattern can be obtained from an 'observation' window located at the end of the prompts. Drawing on this insight, SnapKV automatically compresses KV caches by selecting clustered important KV positions for each attention head. Our approach significantly reduces the growing computational overhead and memory footprint when processing long input sequences. Specifically, SnapKV achieves a consistent decoding speed with a 3.6x increase in generation speed and an 8.2x enhancement in memory efficiency compared to the baseline when processing inputs of 16K tokens. At the same time, it maintains comparable performance to the baseline models across 16 long sequence datasets. Moreover, SnapKV can process up to 380K context tokens on a single A100-80GB GPU using HuggingFace implementation with minor changes, exhibiting only a negligible accuracy drop in the Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's potential for practical applications.","url_abs":"https://arxiv.org/abs/2404.14469v2","url_pdf":"https://arxiv.org/pdf/2404.14469v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"snapkv-llm-knows-what-you-are-looking-for","repo_url":"https://github.com/fasterdecoding/snapkv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"16k","task_name":"16k"},{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.14469","atlas_url":"https://app.syntology.ai/?focus=2404.14469","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.14469"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fasterdecoding/snapkv","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran":3,"ran_honours":1,"ran_draft_wrong":2,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"snapkv/monkeypatch/snapkv_utils.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/snapkv/monkeypatch/snapkv_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"ead90df220ef482e","entry":"build_chat","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"experiments/LongBench/pred_snap.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/experiments/LongBench/pred_snap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ead90df220ef482e"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"experiments/LongBench/metrics.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/experiments/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"experiments/LongBench/metrics.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/experiments/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"experiments/LongBench/metrics.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/experiments/LongBench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"db40c0387e20158c","entry":"parse_args","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"experiments/LongBench/pred_snap.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/experiments/LongBench/pred_snap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db40c0387e20158c"}},{"code_sha256_prefix":"4489113b536ca6eb","entry":"post_process","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"experiments/LongBench/pred_snap.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/experiments/LongBench/pred_snap.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4489113b536ca6eb"}},{"code_sha256_prefix":"7691e90deddd0f79","entry":"prepare_inputs_for_generation_mistral","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"snapkv/monkeypatch/mistral_hijack_4_37.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/snapkv/monkeypatch/mistral_hijack_4_37.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7691e90deddd0f79"}},{"code_sha256_prefix":"d50769dc29155b3b","entry":"prepare_inputs_for_generation_llama","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"snapkv/monkeypatch/llama_hijack_4_37.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/snapkv/monkeypatch/llama_hijack_4_37.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d50769dc29155b3b"}},{"code_sha256_prefix":"5aeb661d1528668c","entry":"prepare_inputs_for_generation_mixtral","repo":"fasterdecoding/snapkv","repo_kind":"official","path":"snapkv/monkeypatch/mixtral_hijack_4_37.py","file_url":"https://github.com/fasterdecoding/snapkv/blob/HEAD/snapkv/monkeypatch/mixtral_hijack_4_37.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5aeb661d1528668c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}