{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-interactive-llm-serving-with-proxy","title":"Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction","arxiv_id":"2404.08509","date":"2024-04-12","proceeding":null,"authors":["Haoran Qiu","Weichao Mao","Archit Patke","Shengkun Cui","Saurabh Jha","Chen Wang","Hubertus Franke","Zbigniew T. Kalbarczyk","Tamer Başar","Ravishankar K. Iyer"],"abstract":"Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.","url_abs":"https://arxiv.org/abs/2404.08509v2","url_pdf":"https://arxiv.org/pdf/2404.08509v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-interactive-llm-serving-with-proxy","repo_url":"https://github.com/james-qiuhaoran/llm-serving-with-proxy-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"blocking","task_name":"Blocking"},{"task_slug":"management","task_name":"Management"},{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.08509","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.08509"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/james-qiuhaoran/llm-serving-with-proxy-models","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"386963aed97c1619","entry":"create_jobs","repo":"james-qiuhaoran/llm-serving-with-proxy-models","repo_kind":"official","path":"model-serving/job.py","file_url":"https://github.com/james-qiuhaoran/llm-serving-with-proxy-models/blob/HEAD/model-serving/job.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"386963aed97c1619"}},{"code_sha256_prefix":"0a577596580d957a","entry":"generate_arrival_list","repo":"james-qiuhaoran/llm-serving-with-proxy-models","repo_kind":"official","path":"model-serving/job.py","file_url":"https://github.com/james-qiuhaoran/llm-serving-with-proxy-models/blob/HEAD/model-serving/job.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a577596580d957a"}},{"code_sha256_prefix":"b9972699195df647","entry":"key_figure","repo":"james-qiuhaoran/llm-serving-with-proxy-models","repo_kind":"official","path":"model-serving/auto_eval_barplot.py","file_url":"https://github.com/james-qiuhaoran/llm-serving-with-proxy-models/blob/HEAD/model-serving/auto_eval_barplot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9972699195df647"}},{"code_sha256_prefix":"98dce69bfe156b53","entry":"set_size","repo":"james-qiuhaoran/llm-serving-with-proxy-models","repo_kind":"official","path":"model-serving/prediction/predictor_overhead_vs_model_serving_latency.py","file_url":"https://github.com/james-qiuhaoran/llm-serving-with-proxy-models/blob/HEAD/model-serving/prediction/predictor_overhead_vs_model_serving_latency.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98dce69bfe156b53"}},{"code_sha256_prefix":"080c0a144aeba7b0","entry":"init_map","repo":"james-qiuhaoran/llm-serving-with-proxy-models","repo_kind":"official","path":"model-serving/auto_eval.py","file_url":"https://github.com/james-qiuhaoran/llm-serving-with-proxy-models/blob/HEAD/model-serving/auto_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"080c0a144aeba7b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}