{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/serverlessllm-locality-enhanced-serverless","title":"ServerlessLLM: Low-Latency Serverless Inference for Large Language Models","arxiv_id":"2401.14351","date":"2024-01-25","proceeding":null,"authors":["Yao Fu","Leyang Xue","Yeqi Huang","Andrei-Octavian Brabete","Dmitrii Ustiugov","Yuvraj Patel","Luo Mai"],"abstract":"This paper presents ServerlessLLM, a distributed system designed to support low-latency serverless inference for Large Language Models (LLMs). By harnessing the substantial near-GPU storage and memory capacities of inference servers, ServerlessLLM achieves effective local checkpoint storage, minimizing the need for remote checkpoint downloads and ensuring efficient checkpoint loading. The design of ServerlessLLM features three core contributions: (i) \\emph{fast multi-tier checkpoint loading}, featuring a new loading-optimized checkpoint format and a multi-tier loading system, fully utilizing the bandwidth of complex storage hierarchies on GPU servers; (ii) \\emph{efficient live migration of LLM inference}, which enables newly initiated inferences to capitalize on local checkpoint storage while ensuring minimal user interruption; and (iii) \\emph{startup-time-optimized model scheduling}, which assesses the locality statuses of checkpoints on each server and schedules the model onto servers that minimize the time to start the inference. Comprehensive evaluations, including microbenchmarks and real-world scenarios, demonstrate that ServerlessLLM dramatically outperforms state-of-the-art serverless systems, reducing latency by 10 - 200X across various LLM inference workloads.","url_abs":"https://arxiv.org/abs/2401.14351v2","url_pdf":"https://arxiv.org/pdf/2401.14351v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"serverlessllm-locality-enhanced-serverless","repo_url":"https://github.com/serverlessllm/serverlessllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.14351","atlas_url":"https://app.syntology.ai/?focus=2401.14351","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.14351"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/serverlessllm/serverlessllm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"66b242d3cebeb8ba","entry":"calculate_stats","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"benchmarks/generate_report.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/benchmarks/generate_report.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66b242d3cebeb8ba"}},{"code_sha256_prefix":"edb868bdca284745","entry":"create_dataframe","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"benchmarks/plot.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/benchmarks/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"edb868bdca284745"}},{"code_sha256_prefix":"2afff7c0bc928466","entry":"generate_text_report","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"benchmarks/generate_report.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/benchmarks/generate_report.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2afff7c0bc928466"}},{"code_sha256_prefix":"0ba2e0c180b4490b","entry":"init_logger","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"sllm/logger.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/sllm/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ba2e0c180b4490b"}},{"code_sha256_prefix":"7003961583dddb9d","entry":"load_results","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"benchmarks/generate_report.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/benchmarks/generate_report.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7003961583dddb9d"}},{"code_sha256_prefix":"1397ecf27bc272fd","entry":"load_results","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"benchmarks/plot.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/benchmarks/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1397ecf27bc272fd"}},{"code_sha256_prefix":"3e05bdb8bd6cf91b","entry":"benchmark_inference","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"benchmarks/benchmark_utils.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/benchmarks/benchmark_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e05bdb8bd6cf91b"}},{"code_sha256_prefix":"215a462b20992a22","entry":"download_lora_adapter","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"sllm/model_downloader.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/sllm/model_downloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"215a462b20992a22"}},{"code_sha256_prefix":"334b8ea78169fbed","entry":"download_transformers_model","repo":"serverlessllm/serverlessllm","repo_kind":"official","path":"sllm/model_downloader.py","file_url":"https://github.com/serverlessllm/serverlessllm/blob/HEAD/sllm/model_downloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"334b8ea78169fbed"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}