{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/attention-score-is-not-all-you-need-for-token","title":"Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters","arxiv_id":"2406.12335","date":"2024-06-18","proceeding":null,"authors":["Zhiyu Guo","Hidetaka Kamigaito","Taro Watanabe"],"abstract":"Scaling the context size of large language models (LLMs) enables them to perform various new tasks, e.g., book summarization. However, the memory cost of the Key and Value (KV) cache in attention significantly limits the practical applications of LLMs. Recent works have explored token pruning for KV cache reduction in LLMs, relying solely on attention scores as a token importance indicator. However, our investigation into value vector norms revealed a notably non-uniform pattern questioning their reliance only on attention scores. Inspired by this, we propose a new method: Value-Aware Token Pruning (VATP) which uses both attention scores and the $ \\ell_{1} $ norm of value vectors to evaluate token importance. Extensive experiments on LLaMA2-7B-chat and Vicuna-v1.5-7B across 16 LongBench tasks demonstrate that VATP outperforms attention-score-only baselines in over 12 tasks, confirming the effectiveness of incorporating value vector norms into token importance evaluation of LLMs.","url_abs":"https://arxiv.org/abs/2406.12335v2","url_pdf":"https://arxiv.org/pdf/2406.12335v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"attention-score-is-not-all-you-need-for-token","repo_url":"https://github.com/guozhiyu/vatp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":null,"task_name":"Book summarization"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.12335","atlas_url":"https://app.syntology.ai/?focus=2406.12335","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12335"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/guozhiyu/vatp","reach":null}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"3613d71443ec2450","entry":"LlamaDynamicNTKScalingRotaryEmbedding","repo":"guozhiyu/vatp","repo_kind":"official","path":"utils_hh/modify_llama.py","file_url":"https://github.com/guozhiyu/vatp/blob/HEAD/utils_hh/modify_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3613d71443ec2450"}},{"code_sha256_prefix":"bc8fe21daaeb0c58","entry":"LlamaLinearScalingRotaryEmbedding","repo":"guozhiyu/vatp","repo_kind":"official","path":"utils_hh/modify_llama.py","file_url":"https://github.com/guozhiyu/vatp/blob/HEAD/utils_hh/modify_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc8fe21daaeb0c58"}},{"code_sha256_prefix":"5151d2a6b2115e55","entry":"LlamaRotaryEmbedding","repo":"guozhiyu/vatp","repo_kind":"official","path":"utils_hh/modify_llama.py","file_url":"https://github.com/guozhiyu/vatp/blob/HEAD/utils_hh/modify_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5151d2a6b2115e55"}},{"code_sha256_prefix":"3189b3bd863c2fad","entry":"LlamaAttention_heavy_hitter","repo":"guozhiyu/vatp","repo_kind":"official","path":"utils_hh/modify_llama.py","file_url":"https://github.com/guozhiyu/vatp/blob/HEAD/utils_hh/modify_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3189b3bd863c2fad"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}