{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fairness-in-serving-large-language-models","title":"Fairness in Serving Large Language Models","arxiv_id":"2401.00588","date":"2023-12-31","proceeding":null,"authors":["Ying Sheng","Shiyi Cao","Dacheng Li","Banghua Zhu","Zhuohan Li","Danyang Zhuo","Joseph E. Gonzalez","Ion Stoica"],"abstract":"High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide range of requests from short chat conversations to long document reading. To ensure that all client requests are processed fairly, most major LLM inference services have request rate limits, to ensure that no client can dominate the request queue. However, this rudimentary notion of fairness also results in under-utilization of the resources and poor client experience when there is spare capacity. While there is a rich literature on fair scheduling, serving LLMs presents new challenges due to their unpredictable request lengths and their unique batching characteristics on parallel accelerators. This paper introduces the definition of LLM serving fairness based on a cost function that accounts for the number of input and output tokens processed. To achieve fairness in serving, we propose a novel scheduling algorithm, the Virtual Token Counter (VTC), a fair scheduler based on the continuous batching mechanism. We prove a 2x tight upper bound on the service difference between two backlogged clients, adhering to the requirement of work-conserving. Through extensive experiments, we demonstrate the superior performance of VTC in ensuring fairness, especially in contrast to other baseline methods, which exhibit shortcomings under various conditions. The reproducible code is available at https://github.com/Ying1123/VTC-artifact","url_abs":"https://arxiv.org/abs/2401.00588v2","url_pdf":"https://arxiv.org/pdf/2401.00588v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fairness-in-serving-large-language-models","repo_url":"https://github.com/ying1123/vtc-artifact","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"fairness-in-serving-large-language-models","repo_url":"https://github.com/s-lora/s-lora","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[{"method_slug":null,"method_name":null}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.00588","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.00588"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/s-lora/s-lora","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ying1123/vtc-artifact","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1},"listed":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6e238d2d144eed9b","entry":"benchmark","repo":"s-lora/s-lora","repo_kind":"listed","path":"benchmarks/run_exp.py","file_url":"https://github.com/s-lora/s-lora/blob/HEAD/benchmarks/run_exp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e238d2d144eed9b"}},{"code_sha256_prefix":"cf380cfb6bd64b21","entry":"dummy_prompt","repo":"ying1123/vtc-artifact","repo_kind":"official","path":"benchmarks/trace.py","file_url":"https://github.com/ying1123/vtc-artifact/blob/HEAD/benchmarks/trace.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf380cfb6bd64b21"}},{"code_sha256_prefix":"2bf66d26ba8a4548","entry":"generate_requests","repo":"ying1123/vtc-artifact","repo_kind":"official","path":"benchmarks/trace.py","file_url":"https://github.com/ying1123/vtc-artifact/blob/HEAD/benchmarks/trace.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2bf66d26ba8a4548"}},{"code_sha256_prefix":"5870f80ad0e8422b","entry":"get_lora_config_json","repo":"s-lora/s-lora","repo_kind":"listed","path":"slora/mprophet/lora_config.py","file_url":"https://github.com/s-lora/s-lora/blob/HEAD/slora/mprophet/lora_config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5870f80ad0e8422b"}},{"code_sha256_prefix":"3719da4f729ce737","entry":"send_request","repo":"s-lora/s-lora","repo_kind":"listed","path":"benchmarks/run_exp.py","file_url":"https://github.com/s-lora/s-lora/blob/HEAD/benchmarks/run_exp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3719da4f729ce737"}},{"code_sha256_prefix":"a027792a13b85ec3","entry":"to_dict","repo":"s-lora/s-lora","repo_kind":"listed","path":"benchmarks/exp_suite.py","file_url":"https://github.com/s-lora/s-lora/blob/HEAD/benchmarks/exp_suite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a027792a13b85ec3"}},{"code_sha256_prefix":"ecdd469e9c7b7443","entry":"to_dict","repo":"ying1123/vtc-artifact","repo_kind":"official","path":"fair_bench/exp_suite.py","file_url":"https://github.com/ying1123/vtc-artifact/blob/HEAD/fair_bench/exp_suite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ecdd469e9c7b7443"}},{"code_sha256_prefix":"1fb624c4bbfa4098","entry":"to_tuple","repo":"s-lora/s-lora","repo_kind":"listed","path":"benchmarks/exp_suite.py","file_url":"https://github.com/s-lora/s-lora/blob/HEAD/benchmarks/exp_suite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fb624c4bbfa4098"}},{"code_sha256_prefix":"98027d2de0638849","entry":"to_tuple","repo":"ying1123/vtc-artifact","repo_kind":"official","path":"fair_bench/exp_suite.py","file_url":"https://github.com/ying1123/vtc-artifact/blob/HEAD/fair_bench/exp_suite.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98027d2de0638849"}},{"code_sha256_prefix":"03e5db79fc455665","entry":"get_all_suites","repo":"s-lora/s-lora","repo_kind":"listed","path":"benchmarks/exp_suite.py","file_url":"https://github.com/s-lora/s-lora/blob/HEAD/benchmarks/exp_suite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"03e5db79fc455665"}},{"code_sha256_prefix":"3b155d666ec2f922","entry":"get_all_suites","repo":"ying1123/vtc-artifact","repo_kind":"official","path":"fair_bench/exp_suite.py","file_url":"https://github.com/ying1123/vtc-artifact/blob/HEAD/fair_bench/exp_suite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b155d666ec2f922"}},{"code_sha256_prefix":"72eaf41bbc94d98b","entry":"get_peak_mem","repo":"s-lora/s-lora","repo_kind":"listed","path":"benchmarks/run_exp.py","file_url":"https://github.com/s-lora/s-lora/blob/HEAD/benchmarks/run_exp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72eaf41bbc94d98b"}},{"code_sha256_prefix":"ceccd6265c698888","entry":"get_real_requests","repo":"ying1123/vtc-artifact","repo_kind":"official","path":"benchmarks/trace.py","file_url":"https://github.com/ying1123/vtc-artifact/blob/HEAD/benchmarks/trace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ceccd6265c698888"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}