{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lserve-efficient-long-sequence-llm-serving","title":"LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention","arxiv_id":"2502.14866","date":"2025-02-20","proceeding":null,"authors":["Shang Yang","Junxian Guo","Haotian Tang","Qinghao Hu","Guangxuan Xiao","Jiaming Tang","Yujun Lin","Zhijian Liu","Yao Lu","Song Han"],"abstract":"Large language models (LLMs) have shown remarkable potential in processing long sequences and complex reasoning tasks, yet efficiently serving these models remains challenging due to the quadratic computational complexity of attention in the prefilling stage and the large memory footprint of the KV cache in the decoding stage. To address these issues, we introduce LServe, an efficient system that accelerates long-sequence LLM serving via hybrid sparse attention. This method unifies different hardware-friendly, structured sparsity patterns for both prefilling and decoding attention into a single framework, where computations on less important tokens are skipped block-wise. LServe demonstrates the compatibility of static and dynamic sparsity in long-context LLM attention. This design enables multiplicative speedups by combining these optimizations. Specifically, we convert half of the attention heads to nearly free streaming heads in both the prefilling and decoding stages. Additionally, we find that only a constant number of KV pages is required to preserve long-context and reasoning capabilities, irrespective of context length. We then design a hierarchical KV page selection policy that dynamically prunes KV pages based on query-centric similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is released at https://github.com/mit-han-lab/omniserve.","url_abs":"https://arxiv.org/abs/2502.14866v2","url_pdf":"https://arxiv.org/pdf/2502.14866v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lserve-efficient-long-sequence-llm-serving","repo_url":"https://github.com/mit-han-lab/omniserve","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"lserve-efficient-long-sequence-llm-serving","repo_url":"https://github.com/mit-han-lab/qserve","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.14866","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.14866"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-han-lab/qserve","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-han-lab/omniserve","reach":null}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3b545dce3ebb9392","entry":"get_conv_template","repo":"mit-han-lab/qserve","repo_kind":"listed","path":"omniserve/conversation.py","file_url":"https://github.com/mit-han-lab/qserve/blob/HEAD/omniserve/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b545dce3ebb9392"}},{"code_sha256_prefix":"6d66d1587bbcf675","entry":"get_conv_template_name","repo":"mit-han-lab/qserve","repo_kind":"listed","path":"omniserve/conversation.py","file_url":"https://github.com/mit-han-lab/qserve/blob/HEAD/omniserve/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d66d1587bbcf675"}},{"code_sha256_prefix":"4e6200c3c6f4170a","entry":"init_logger","repo":"mit-han-lab/qserve","repo_kind":"listed","path":"omniserve/logger.py","file_url":"https://github.com/mit-han-lab/qserve/blob/HEAD/omniserve/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e6200c3c6f4170a"}},{"code_sha256_prefix":"f8b39512a4a5d57e","entry":"read_haystack_files","repo":"mit-han-lab/qserve","repo_kind":"listed","path":"lserve_e2e_generation.py","file_url":"https://github.com/mit-han-lab/qserve/blob/HEAD/lserve_e2e_generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8b39512a4a5d57e"}},{"code_sha256_prefix":"5d2dee48f3c9af5e","entry":"sparse_attn_init","repo":"mit-han-lab/omniserve","repo_kind":"official","path":"omniserve/attn_config.py","file_url":"https://github.com/mit-han-lab/omniserve/blob/HEAD/omniserve/attn_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d2dee48f3c9af5e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}