{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-08305","title":"LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems","arxiv_id":"2605.08305","date":"2026-05-08","proceeding":null,"authors":["Siyu Wu","Yulong Ye","Zezhen Xiang","Pengzhou Chen","Gangda Xiong","Tao Chen"],"abstract":"Large Language Model (LLM) systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing benchmarks. This paper presents the first (live) benchmark suite and datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench, covering data related to the inference objective values of hyperparameter configurations profiled from running the LLM systems. Currently, LLMSYS-HPOBench contains 364,450 hyperparameter configurations with a dimensionality of 12-23, 3-5 dimensions of fidelity factor leading to 932 settings, 3-9 inference objective metrics, and 2-10 cost metrics, together with generated logs from measuring the LLM systems. What we seek to advocate is not only a revalidation of the existing HPO algorithms over the frontier LLM systems, but also to provide an evolving platform for the AutoML community to explore new directions of research in this regard. The benchmark suite has been made available at: https://github.com/ideas-labo/llmsys-hpobench","url_abs":"https://arxiv.org/abs/2605.08305","url_pdf":"https://arxiv.org/pdf/2605.08305","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.08305","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.08305"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ideas-labo/llmsys-hpobench","reach":null}],"summary":{"ran":18},"by_repo_kind":{"found_in_text":{"samples":18,"ran":18,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"9d8be5dd0f1dc88d","entry":"async_request_deepspeed_mii","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/benchmark/backend_request_func.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/benchmark/backend_request_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9d8be5dd0f1dc88d"}},{"code_sha256_prefix":"4eb1fa1a6a225d17","entry":"async_request_tgi","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/benchmark/backend_request_func.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/benchmark/backend_request_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4eb1fa1a6a225d17"}},{"code_sha256_prefix":"f273b25928123734","entry":"async_request_trt_llm","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/benchmark/backend_request_func.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/benchmark/backend_request_func.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f273b25928123734"}},{"code_sha256_prefix":"2fc4b9804acb5652","entry":"calculate_lexical_answer_correctness","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/RAG/lightrag_sampler/lightrag_run.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/RAG/lightrag_sampler/lightrag_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2fc4b9804acb5652"}},{"code_sha256_prefix":"636bdb2bc120aa1f","entry":"calculate_statistics","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/RAG/lightrag_sampler/fidelity_analyze.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/RAG/lightrag_sampler/fidelity_analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"636bdb2bc120aa1f"}},{"code_sha256_prefix":"c88e3e5114a284ab","entry":"calculate_text_similarity","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/sampler.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c88e3e5114a284ab"}},{"code_sha256_prefix":"6709a1a771746bd3","entry":"count_csv_rows","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"analyze.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6709a1a771746bd3"}},{"code_sha256_prefix":"89db831e06e49114","entry":"find_system_dirs","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"analyze.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"89db831e06e49114"}},{"code_sha256_prefix":"49b45d38085c6641","entry":"get_param_combinations","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/MFSampler.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/MFSampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"49b45d38085c6641"}},{"code_sha256_prefix":"e77c39359a344f02","entry":"is_valid_sequence","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/benchmark/benchmark_dataset.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/benchmark/benchmark_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e77c39359a344f02"}},{"code_sha256_prefix":"4b7849ab3b37efbb","entry":"merge_csv_files","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/RAG/lightrag_sampler/fidelity_analyze.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/RAG/lightrag_sampler/fidelity_analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4b7849ab3b37efbb"}},{"code_sha256_prefix":"cc5b2aa02b874331","entry":"sample_requests_from_dataset","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/benchmark/benchmark_prefix_caching.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/benchmark/benchmark_prefix_caching.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cc5b2aa02b874331"}},{"code_sha256_prefix":"22f76bb747d8133d","entry":"sample_requests_from_random","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/benchmark/benchmark_prefix_caching.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/benchmark/benchmark_prefix_caching.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"22f76bb747d8133d"}},{"code_sha256_prefix":"5110107e8623745d","entry":"sample_sharegpt_requests","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/sampler.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5110107e8623745d"}},{"code_sha256_prefix":"66718c872ef47e16","entry":"sample_sonnet_requests","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/sampler.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"66718c872ef47e16"}},{"code_sha256_prefix":"493b76d45cef9f16","entry":"sample_tokens","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/Engine/vllm_sampler/benchmark/benchmark_prefix_caching.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/Engine/vllm_sampler/benchmark/benchmark_prefix_caching.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"493b76d45cef9f16"}},{"code_sha256_prefix":"7f3295bf2278cfd2","entry":"sort_csv_files_by_fidelity","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"sampler/RAG/lightrag_sampler/fidelity_analyze.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/sampler/RAG/lightrag_sampler/fidelity_analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7f3295bf2278cfd2"}},{"code_sha256_prefix":"a13d50b27efaf37c","entry":"summarize_system","repo":"ideas-labo/llmsys-hpobench","repo_kind":"found_in_text","path":"analyze.py","file_url":"https://github.com/ideas-labo/llmsys-hpobench/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a13d50b27efaf37c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}