{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/layerkv-optimizing-large-language-model","title":"LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management","arxiv_id":"2410.00428","date":"2024-10-01","proceeding":null,"authors":["Yi Xiong","Hao Wu","Changxu Shao","Ziqing Wang","Rui Zhang","Yuhong Guo","Junping Zhao","Ke Zhang","Zhenxuan Pan"],"abstract":"The expanding context windows in large language models (LLMs) have greatly enhanced their capabilities in various applications, but they also introduce significant challenges in maintaining low latency, particularly in Time to First Token (TTFT). This paper identifies that the sharp rise in TTFT as context length increases is predominantly driven by queuing delays, which are caused by the growing demands for GPU Key-Value (KV) cache allocation clashing with the limited availability of KV cache blocks. To address this issue, we propose LayerKV, a simple yet effective plug-in method that effectively reduces TTFT without requiring additional hardware or compromising output performance, while seamlessly integrating with existing parallelism strategies and scheduling techniques. Specifically, LayerKV introduces layer-wise KV block allocation, management, and offloading for fine-grained control over system memory, coupled with an SLO-aware scheduler to optimize overall Service Level Objectives (SLOs). Comprehensive evaluations on representative models, ranging from 7B to 70B parameters, across various GPU configurations, demonstrate that LayerKV improves TTFT latency up to 69x and reduces SLO violation rates by 28.7%, significantly enhancing the user experience.","url_abs":"https://arxiv.org/abs/2410.00428v3","url_pdf":"https://arxiv.org/pdf/2410.00428v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"layerkv-optimizing-large-language-model","repo_url":"https://github.com/intelligent-machine-learning/glake","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"management","task_name":"Management"},{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[{"method_slug":null,"method_name":null}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.00428","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.00428"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/intelligent-machine-learning/glake","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":9},"by_repo_kind":{"listed":{"samples":14,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5109c385e92b727e","entry":"async_request_deepspeed_mii","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/benchmarks/backend_request_func.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/benchmarks/backend_request_func.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5109c385e92b727e"}},{"code_sha256_prefix":"23a62a462a38f854","entry":"async_request_tgi","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/benchmarks/backend_request_func.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/benchmarks/backend_request_func.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23a62a462a38f854"}},{"code_sha256_prefix":"fa581ebc13cb414d","entry":"async_request_trt_llm","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/benchmarks/backend_request_func.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/benchmarks/backend_request_func.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa581ebc13cb414d"}},{"code_sha256_prefix":"5ae630bfc3821b6e","entry":"run_and_read_all","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/collect_env.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/collect_env.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ae630bfc3821b6e"}},{"code_sha256_prefix":"f55439b30be97107","entry":"sample_requests","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/benchmarks/benchmark_throughput.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/benchmarks/benchmark_throughput.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f55439b30be97107"}},{"code_sha256_prefix":"c382c6353e8cccb0","entry":"awq_dequantize","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/vllm/_custom_ops.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/vllm/_custom_ops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c382c6353e8cccb0"}},{"code_sha256_prefix":"f3fcfeaca594c86f","entry":"awq_gemm","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/vllm/_custom_ops.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/vllm/_custom_ops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3fcfeaca594c86f"}},{"code_sha256_prefix":"355e5d19707b7c81","entry":"get_served_model_name","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/vllm/config.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/vllm/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"355e5d19707b7c81"}},{"code_sha256_prefix":"4def2cf22c36f4e8","entry":"gptq_gemm","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/vllm/_custom_ops.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/vllm/_custom_ops.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4def2cf22c36f4e8"}},{"code_sha256_prefix":"6b73fe0480a96ad9","entry":"run","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/collect_env.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/collect_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b73fe0480a96ad9"}},{"code_sha256_prefix":"ce7a62e37fe78c0d","entry":"run_and_parse_first_match","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/collect_env.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/collect_env.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce7a62e37fe78c0d"}},{"code_sha256_prefix":"0745e2813903bd8d","entry":"run_hf","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/benchmarks/benchmark_throughput.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/benchmarks/benchmark_throughput.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0745e2813903bd8d"}},{"code_sha256_prefix":"f44718a5d7ec296f","entry":"run_vllm","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/benchmarks/benchmark_throughput.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/benchmarks/benchmark_throughput.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f44718a5d7ec296f"}},{"code_sha256_prefix":"8abfc7dfff3ebb06","entry":"sample_requests","repo":"intelligent-machine-learning/glake","repo_kind":"listed","path":"GLakeServe/benchmarks/benchmark_serving.py","file_url":"https://github.com/intelligent-machine-learning/glake/blob/HEAD/GLakeServe/benchmarks/benchmark_serving.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8abfc7dfff3ebb06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}