{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/loongserve-efficiently-serving-long-context","title":"LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism","arxiv_id":"2404.09526","date":"2024-04-15","proceeding":null,"authors":["Bingyang Wu","Shengyu Liu","Yinmin Zhong","Peng Sun","Xuanzhe Liu","Xin Jin"],"abstract":"The context window of large language models (LLMs) is rapidly increasing, leading to a huge variance in resource usage between different requests as well as between different phases of the same request. Restricted by static parallelism strategies, existing LLM serving systems cannot efficiently utilize the underlying resources to serve variable-length requests in different phases. To address this problem, we propose a new parallelism paradigm, elastic sequence parallelism (ESP), to elastically adapt to the variance between different requests and phases. Based on ESP, we design and build LoongServe, an LLM serving system that (1) improves computation efficiency by elastically adjusting the degree of parallelism in real-time, (2) improves communication efficiency by reducing key-value cache migration overhead and overlapping partial decoding communication with computation, and (3) improves GPU memory efficiency by reducing key-value cache fragmentation across instances. Our evaluation under diverse real-world datasets shows that LoongServe improves the maximum throughput by up to 3.85$\\times$ compared to the chunked prefill and 5.81$\\times$ compared to the prefill-decoding disaggregation.","url_abs":"https://arxiv.org/abs/2404.09526v2","url_pdf":"https://arxiv.org/pdf/2404.09526v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"loongserve-efficiently-serving-long-context","repo_url":"https://github.com/LoongServe/LoongServe","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"dilated-convolution","method_name":"Dilated Convolution"},{"method_slug":"esp","method_name":"ESP"},{"method_slug":"fragmentation","method_name":"Fragmentation"},{"method_slug":"hierarchical-feature-fusion","method_name":"Hierarchical Feature Fusion"},{"method_slug":"pointwise-convolution","method_name":"Pointwise Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.09526","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.09526"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LoongServe/LoongServe","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ced3ee0f9e38c4cf","entry":"alloc_can_use_network_port","repo":"LoongServe/LoongServe","repo_kind":"official","path":"loongserve/utils/net_utils.py","file_url":"https://github.com/LoongServe/LoongServe/blob/HEAD/loongserve/utils/net_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ced3ee0f9e38c4cf"}},{"code_sha256_prefix":"b710e60c3d04dc58","entry":"build_prompt","repo":"LoongServe/LoongServe","repo_kind":"official","path":"loongserve/longserve_server/build_prompt.py","file_url":"https://github.com/LoongServe/LoongServe/blob/HEAD/loongserve/longserve_server/build_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b710e60c3d04dc58"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}