{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/expobench-benchmarking-surrogate-based","title":"EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions","arxiv_id":"2106.04618","date":"2021-06-08","proceeding":null,"authors":["Laurens Bliek","Arthur Guijt","Rickard Karlsson","Sicco Verwer","Mathijs de Weerdt"],"abstract":"Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions and to give substantial advice on which method to use when. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. We also provide rules of thumb for which surrogate algorithm to use in which situation. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the performance of the six algorithms on all evaluated problem instances. This results in a unique new dataset that lowers the bar for researching new methods as the number of expensive evaluations required for comparison is significantly reduced.","url_abs":"https://arxiv.org/abs/2106.04618v2","url_pdf":"https://arxiv.org/pdf/2106.04618v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"expobench-benchmarking-surrogate-based","repo_url":"https://github.com/AlgTUDelft/ExpensiveOptimBenchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"bayesian-optimisation","task_name":"Bayesian Optimisation"},{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04618","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04618"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AlgTUDelft/ExpensiveOptimBenchmark","reach":null}],"summary":{"ran_fixture":1,"ran_honours":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d5dab9290c05bed","entry":"construct_tsp","repo":"AlgTUDelft/ExpensiveOptimBenchmark","repo_kind":"official","path":"expensiveoptimbenchmark/run_experiment.py","file_url":"https://github.com/AlgTUDelft/ExpensiveOptimBenchmark/blob/HEAD/expensiveoptimbenchmark/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d5dab9290c05bed"}},{"code_sha256_prefix":"c4b2c4813c0c5515","entry":"parse_numerical_range","repo":"AlgTUDelft/ExpensiveOptimBenchmark","repo_kind":"official","path":"expensiveoptimbenchmark/run_experiment.py","file_url":"https://github.com/AlgTUDelft/ExpensiveOptimBenchmark/blob/HEAD/expensiveoptimbenchmark/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4b2c4813c0c5515"}},{"code_sha256_prefix":"ccec78f73066da9d","entry":"parse_numerical_ranges","repo":"AlgTUDelft/ExpensiveOptimBenchmark","repo_kind":"official","path":"expensiveoptimbenchmark/run_experiment.py","file_url":"https://github.com/AlgTUDelft/ExpensiveOptimBenchmark/blob/HEAD/expensiveoptimbenchmark/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccec78f73066da9d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}