{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/splitwise-efficient-generative-llm-inference","title":"Splitwise: Efficient generative LLM inference using phase splitting","arxiv_id":"2311.18677","date":"2023-11-30","proceeding":null,"authors":["Pratyush Patel","Esha Choukse","Chaojie Zhang","Aashaka Shah","Íñigo Goiri","Saeed Maleki","Ricardo Bianchini"],"abstract":"Recent innovations in generative large language models (LLMs) have made their applications and use-cases ubiquitous. This has led to large-scale deployments of these models, using complex, expensive, and power-hungry AI accelerators, most commonly GPUs. These developments make LLM inference efficiency an important challenge. Based on our extensive characterization, we find that there are two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics. Despite state-of-the-art batching and scheduling, the token generation phase underutilizes compute resources. Specifically, unlike compute-intensive prompt computation phases, token generation phases do not require the compute capability of the latest GPUs, and can be run with lower power and cost. With Splitwise, we propose splitting the two phases of a LLM inference request on to separate machines. This allows us to use hardware that is well-suited for each phase, and provision resources independently per phase. However, splitting an inference request across machines requires state transfer from the machine running prompt computation over to the machine generating tokens. We implement and optimize this state transfer using the fast back-plane interconnects available in today's GPU clusters. We use the Splitwise technique to design LLM inference clusters using the same or different types of machines for the prompt computation and token generation phases. Our clusters are optimized for three key objectives: throughput, cost, and power. In particular, we show that we can achieve 1.4x higher throughput at 20% lower cost than current designs. Alternatively, we can achieve 2.35x more throughput with the same cost and power budgets.","url_abs":"https://arxiv.org/abs/2311.18677v2","url_pdf":"https://arxiv.org/pdf/2311.18677v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"splitwise-efficient-generative-llm-inference","repo_url":"https://github.com/microsoft/mscclpp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"splitwise-efficient-generative-llm-inference","repo_url":"https://github.com/Azure/AzurePublicDataset","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"paper_slug":"splitwise-efficient-generative-llm-inference","repo_url":"https://github.com/Mutinifni/splitwise-sim","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"splitwise-efficient-generative-llm-inference","repo_url":"https://github.com/efeslab/Nanoflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.18677","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.18677"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/efeslab/Nanoflow","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/mscclpp","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Azure/AzurePublicDataset","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mutinifni/splitwise-sim","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":2},"named_in_paper":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"ad9d55a26906b5ea","entry":"exponential_func","repo":"Azure/AzurePublicDataset","repo_kind":"official","path":"analysis/GreenSKU-Framework/src/derate_curve.py","file_url":"https://github.com/Azure/AzurePublicDataset/blob/HEAD/analysis/GreenSKU-Framework/src/derate_curve.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ad9d55a26906b5ea"}},{"code_sha256_prefix":"265acc16bd2f6830","entry":"get_dict_opex","repo":"Azure/AzurePublicDataset","repo_kind":"official","path":"analysis/GreenSKU-Framework/src/carbon_model.py","file_url":"https://github.com/Azure/AzurePublicDataset/blob/HEAD/analysis/GreenSKU-Framework/src/carbon_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"265acc16bd2f6830"}},{"code_sha256_prefix":"5f01c092abff1074","entry":"get_opex","repo":"Azure/AzurePublicDataset","repo_kind":"official","path":"analysis/GreenSKU-Framework/src/carbon_model.py","file_url":"https://github.com/Azure/AzurePublicDataset/blob/HEAD/analysis/GreenSKU-Framework/src/carbon_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5f01c092abff1074"}},{"code_sha256_prefix":"4289a7837d69706c","entry":"get_opex_from_dict","repo":"Azure/AzurePublicDataset","repo_kind":"official","path":"analysis/GreenSKU-Framework/src/carbon_model.py","file_url":"https://github.com/Azure/AzurePublicDataset/blob/HEAD/analysis/GreenSKU-Framework/src/carbon_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4289a7837d69706c"}},{"code_sha256_prefix":"02639e1a8ca8fe63","entry":"human_readable_size","repo":"microsoft/mscclpp","repo_kind":"official","path":"python/mscclpp_benchmark/allreduce_bench.py","file_url":"https://github.com/microsoft/mscclpp/blob/HEAD/python/mscclpp_benchmark/allreduce_bench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02639e1a8ca8fe63"}},{"code_sha256_prefix":"75e8bf267d221c4f","entry":"quadratic_func","repo":"Azure/AzurePublicDataset","repo_kind":"official","path":"analysis/GreenSKU-Framework/src/derate_curve.py","file_url":"https://github.com/Azure/AzurePublicDataset/blob/HEAD/analysis/GreenSKU-Framework/src/derate_curve.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"75e8bf267d221c4f"}},{"code_sha256_prefix":"506db674bec6177c","entry":"generate_samples","repo":"Mutinifni/splitwise-sim","repo_kind":"named_in_paper","path":"generate_trace.py","file_url":"https://github.com/Mutinifni/splitwise-sim/blob/HEAD/generate_trace.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"506db674bec6177c"}},{"code_sha256_prefix":"4484b5133c7d2676","entry":"generate_trace","repo":"Mutinifni/splitwise-sim","repo_kind":"named_in_paper","path":"generate_trace.py","file_url":"https://github.com/Mutinifni/splitwise-sim/blob/HEAD/generate_trace.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4484b5133c7d2676"}},{"code_sha256_prefix":"000bada9d64375ba","entry":"get_exponential_scale","repo":"Mutinifni/splitwise-sim","repo_kind":"named_in_paper","path":"generate_trace.py","file_url":"https://github.com/Mutinifni/splitwise-sim/blob/HEAD/generate_trace.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"000bada9d64375ba"}},{"code_sha256_prefix":"a902f7f41d56887f","entry":"get_processors_power","repo":"Mutinifni/splitwise-sim","repo_kind":"named_in_paper","path":"power_model.py","file_url":"https://github.com/Mutinifni/splitwise-sim/blob/HEAD/power_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a902f7f41d56887f"}},{"code_sha256_prefix":"74353330e86481bb","entry":"is_torch_tensor","repo":"microsoft/mscclpp","repo_kind":"official","path":"python/mscclpp/utils.py","file_url":"https://github.com/microsoft/mscclpp/blob/HEAD/python/mscclpp/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74353330e86481bb"}},{"code_sha256_prefix":"70c2b2f7c128e238","entry":"linear_func","repo":"Azure/AzurePublicDataset","repo_kind":"official","path":"analysis/GreenSKU-Framework/src/derate_curve.py","file_url":"https://github.com/Azure/AzurePublicDataset/blob/HEAD/analysis/GreenSKU-Framework/src/derate_curve.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"70c2b2f7c128e238"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}