{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prefixkv-adaptive-prefix-kv-cache-is-what","title":"PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation","arxiv_id":"2412.03409","date":"2024-12-04","proceeding":null,"authors":["Ao Wang","Hui Chen","Jianchao Tan","Kefeng Zhang","Xunliang Cai","Zijia Lin","Jungong Han","Guiguang Ding"],"abstract":"Recently, large vision-language models (LVLMs) have rapidly gained popularity for their strong generation and reasoning capabilities given diverse multimodal inputs. However, these models incur significant computational and memory overhead during inference, which greatly hinders the efficient deployment in practical scenarios. The extensive key-value (KV) cache, necessitated by the lengthy input and output sequences, notably contributes to the high inference cost. Based on this, recent works have investigated ways to reduce the KV cache size for higher efficiency. Although effective, they generally overlook the distinct importance distributions of KV vectors across layers and maintain the same cache size for each layer during the next token prediction. This results in the significant contextual information loss for certain layers, leading to notable performance decline. To address this, we present PrefixKV. It reframes the challenge of determining KV cache sizes for all layers into the task of searching for the optimal global prefix configuration. With an adaptive layer-wise KV retention recipe based on binary search, the maximum contextual information can thus be preserved in each layer, facilitating the generation. Extensive experiments demonstrate that our method achieves the state-of-the-art performance compared with others. It exhibits superior inference efficiency and generation quality trade-offs, showing promising potential for practical applications. Code is available at \\url{https://github.com/THU-MIG/PrefixKV}.","url_abs":"https://arxiv.org/abs/2412.03409v2","url_pdf":"https://arxiv.org/pdf/2412.03409v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prefixkv-adaptive-prefix-kv-cache-is-what","repo_url":"https://github.com/THU-MIG/PrefixKV","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2412.03409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.03409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/THU-MIG/PrefixKV","reach":null}],"summary":{"ran_honours":1,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"727f447af4c6cf17","entry":"slice1d","repo":"thu-mig/prefixkv","repo_kind":"official","path":"prefixkv.py","file_url":"https://github.com/thu-mig/prefixkv/blob/HEAD/prefixkv.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"727f447af4c6cf17"}},{"code_sha256_prefix":"349c21c206f3e211","entry":"slice3d","repo":"thu-mig/prefixkv","repo_kind":"official","path":"prefixkv.py","file_url":"https://github.com/thu-mig/prefixkv/blob/HEAD/prefixkv.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"349c21c206f3e211"}},{"code_sha256_prefix":"ef0f018176d5ec13","entry":"PrefixKV","repo":"THU-MIG/PrefixKV","repo_kind":"official","path":"prefixkv.py","file_url":"https://github.com/THU-MIG/PrefixKV/blob/HEAD/prefixkv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef0f018176d5ec13"}},{"code_sha256_prefix":"bec4a2080b4fd55d","entry":"obtain_cdf_num","repo":"THU-MIG/PrefixKV","repo_kind":"official","path":"prefixkv.py","file_url":"https://github.com/THU-MIG/PrefixKV/blob/HEAD/prefixkv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bec4a2080b4fd55d"}},{"code_sha256_prefix":"38bd668b3d930f03","entry":"slice2d","repo":"thu-mig/prefixkv","repo_kind":"official","path":"prefixkv.py","file_url":"https://github.com/thu-mig/prefixkv/blob/HEAD/prefixkv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38bd668b3d930f03"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}