{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-17000","title":"BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks","arxiv_id":"2605.17000","date":"2026-05-16","proceeding":null,"authors":["Ruth Wan Theng Chew","Zhiliang Chen","Apivich Hemachandra","Bryan Kian Hsiang Low"],"abstract":"Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is often approached heuristically in practice, leading to potentially suboptimal outcomes. By framing them as noisy, expensive, and derivative-free optimization problems, Bayesian optimization (BO) and other black-box optimization (BBO) methods offer a promising yet underexplored direction for principled, sample-efficient methods. However, LLM training and inference costs are prohibitively high for most of the BBO research community, and new methods are often only evaluated on synthetic test functions and small-scale datasets that fail to capture the challenges of modern LLM optimization problems. This impedes the development of BBO methods and makes it difficult to assess their effectiveness on modern LLM tasks. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT is released at https://github.com/chewwt/bolt. BoLT covers broad and well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, and high-dimensional search spaces. Each problem in BoLT is grounded in real experimental data and made fully reproducible and accessible through lightweight surrogate models fitted to the results of thousands of real LLM experiments. We benchmark BoLT against an extensive range of BO and BBO methods, showing that selected BO methods consistently outperform others across tasks and highlighting gaps in existing BBO methods on LLM tasks, underscoring the need to modernize benchmarks for the BBO community.","url_abs":"https://arxiv.org/abs/2605.17000","url_pdf":"https://arxiv.org/pdf/2605.17000","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.17000","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.17000"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/chewwt/bolt","reach":null}],"summary":{"ran":1,"ran_honours":1,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":7,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf6ff856c2a15b6e","entry":"FeatureNet","repo":"chewwt/bolt","repo_kind":"found_in_text","path":"bolt/problems/hpo.py","file_url":"https://github.com/chewwt/bolt/blob/HEAD/bolt/problems/hpo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf6ff856c2a15b6e"}},{"code_sha256_prefix":"15d6323933269659","entry":"unstandardize_y","repo":"chewwt/bolt","repo_kind":"found_in_text","path":"bolt/problems/hpo.py","file_url":"https://github.com/chewwt/bolt/blob/HEAD/bolt/problems/hpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15d6323933269659"}},{"code_sha256_prefix":"7e28428ff11a95f2","entry":"Function","repo":"chewwt/bolt","repo_kind":"found_in_text","path":"bolt/problems/hpo.py","file_url":"https://github.com/chewwt/bolt/blob/HEAD/bolt/problems/hpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e28428ff11a95f2"}},{"code_sha256_prefix":"5a038c9080c4264e","entry":"HPO","repo":"chewwt/bolt","repo_kind":"found_in_text","path":"bolt/problems/hpo.py","file_url":"https://github.com/chewwt/bolt/blob/HEAD/bolt/problems/hpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a038c9080c4264e"}},{"code_sha256_prefix":"136417067b4cea14","entry":"LLMTestProblem","repo":"chewwt/bolt","repo_kind":"found_in_text","path":"bolt/problems/hpo.py","file_url":"https://github.com/chewwt/bolt/blob/HEAD/bolt/problems/hpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"136417067b4cea14"}},{"code_sha256_prefix":"b84d3823cf21da05","entry":"MLPFunction","repo":"chewwt/bolt","repo_kind":"found_in_text","path":"bolt/problems/hpo.py","file_url":"https://github.com/chewwt/bolt/blob/HEAD/bolt/problems/hpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b84d3823cf21da05"}},{"code_sha256_prefix":"a8c992225cc55ae6","entry":"pull_info_from_hf_hub","repo":"chewwt/bolt","repo_kind":"found_in_text","path":"bolt/problems/hpo.py","file_url":"https://github.com/chewwt/bolt/blob/HEAD/bolt/problems/hpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8c992225cc55ae6"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}