{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/taming-throughput-latency-tradeoff-in-llm","title":"Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve","arxiv_id":"2403.02310","date":"2024-03-04","proceeding":null,"authors":["Amey Agrawal","Nitin Kedia","Ashish Panwar","Jayashree Mohan","Nipun Kwatra","Bhargav S. Gulavani","Alexey Tumanov","Ramachandran Ramjee"],"abstract":"Each LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt. In contrast, decode iterations have low latency but also low compute utilization because a decode iteration processes only a single token per request. This makes batching highly effective for decodes and consequently for overall throughput. However, batching multiple requests leads to an interleaving of prefill and decode iterations which makes it challenging to achieve both high throughput and low latency. We introduce an efficient LLM inference scheduler, Sarathi-Serve, to address this throughput-latency tradeoff. Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes. Stall-free scheduling unlocks the opportunity to improve throughput with large batch sizes while minimizing the effect of batching on latency. Furthermore, uniform batches in Sarathi-Serve ameliorate the imbalance between iterations resulting in minimal pipeline bubbles. Our techniques yield significant improvements in inference performance across models and hardware under tail latency constraints. For Mistral-7B on single A100 GPUs, we achieve 2.6x higher serving capacity and up to 3.7x higher serving capacity for the Yi-34B model on two A100 GPUs as compared to vLLM. When used with pipeline parallelism on Falcon-180B, Sarathi-Serve provides up to 5.6x gain in the end-to-end serving capacity. The source code for Sarathi-Serve is available at https://github.com/microsoft/sarathi-serve.","url_abs":"https://arxiv.org/abs/2403.02310v3","url_pdf":"https://arxiv.org/pdf/2403.02310v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"taming-throughput-latency-tradeoff-in-llm","repo_url":"https://github.com/microsoft/sarathi-serve","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.02310","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.02310"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/sarathi-serve","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"68ee0f6ef8eb5004","entry":"convert_pyslice_to_tensor","repo":"microsoft/sarathi-serve","repo_kind":"official","path":"sarathi/model_executor/weight_utils.py","file_url":"https://github.com/microsoft/sarathi-serve/blob/HEAD/sarathi/model_executor/weight_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68ee0f6ef8eb5004"}},{"code_sha256_prefix":"75f9e29d6669a2b6","entry":"get_lock","repo":"microsoft/sarathi-serve","repo_kind":"official","path":"sarathi/model_executor/weight_utils.py","file_url":"https://github.com/microsoft/sarathi-serve/blob/HEAD/sarathi/model_executor/weight_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75f9e29d6669a2b6"}},{"code_sha256_prefix":"197a0f338024eca9","entry":"init_logger","repo":"microsoft/sarathi-serve","repo_kind":"official","path":"sarathi/logger.py","file_url":"https://github.com/microsoft/sarathi-serve/blob/HEAD/sarathi/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"197a0f338024eca9"}},{"code_sha256_prefix":"43e0fdf0b69f6fd5","entry":"pad_to_alignment","repo":"microsoft/sarathi-serve","repo_kind":"official","path":"sarathi/model_executor/utils.py","file_url":"https://github.com/microsoft/sarathi-serve/blob/HEAD/sarathi/model_executor/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43e0fdf0b69f6fd5"}},{"code_sha256_prefix":"9dcee4f7df3d2300","entry":"pad_to_max","repo":"microsoft/sarathi-serve","repo_kind":"official","path":"sarathi/model_executor/utils.py","file_url":"https://github.com/microsoft/sarathi-serve/blob/HEAD/sarathi/model_executor/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dcee4f7df3d2300"}},{"code_sha256_prefix":"04ad4cad55976bc7","entry":"prepare_hf_model_weights","repo":"microsoft/sarathi-serve","repo_kind":"official","path":"sarathi/model_executor/weight_utils.py","file_url":"https://github.com/microsoft/sarathi-serve/blob/HEAD/sarathi/model_executor/weight_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"04ad4cad55976bc7"}},{"code_sha256_prefix":"2748c310c15773c2","entry":"round_up_to_multiple","repo":"microsoft/sarathi-serve","repo_kind":"official","path":"sarathi/model_executor/utils.py","file_url":"https://github.com/microsoft/sarathi-serve/blob/HEAD/sarathi/model_executor/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2748c310c15773c2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}