{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bayesian-optimization-is-superior-to-random","title":"Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020","arxiv_id":"2104.10201","date":"2021-04-20","proceeding":null,"authors":["Ryan Turner","David Eriksson","Michael McCourt","Juha Kiili","Eero Laaksonen","Zhen Xu","Isabelle Guyon"],"abstract":"This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.","url_abs":"https://arxiv.org/abs/2104.10201v2","url_pdf":"https://arxiv.org/pdf/2104.10201v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bayesian-optimization-is-superior-to-random","repo_url":"https://github.com/facebookresearch/nevergrad","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2104.10201","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.10201"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/nevergrad","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"named_in_paper":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"03e56941d916d955","entry":"gym_budget_modifier","repo":"facebookresearch/nevergrad","repo_kind":"named_in_paper","path":"nevergrad/benchmark/gymexperiments.py","file_url":"https://github.com/facebookresearch/nevergrad/blob/HEAD/nevergrad/benchmark/gymexperiments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"03e56941d916d955"}},{"code_sha256_prefix":"c91903c995874db6","entry":"gym_optimizer_modifier","repo":"facebookresearch/nevergrad","repo_kind":"named_in_paper","path":"nevergrad/benchmark/gymexperiments.py","file_url":"https://github.com/facebookresearch/nevergrad/blob/HEAD/nevergrad/benchmark/gymexperiments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c91903c995874db6"}},{"code_sha256_prefix":"ee04d3a616dfbeb7","entry":"gym_problem_modifier","repo":"facebookresearch/nevergrad","repo_kind":"named_in_paper","path":"nevergrad/benchmark/gymexperiments.py","file_url":"https://github.com/facebookresearch/nevergrad/blob/HEAD/nevergrad/benchmark/gymexperiments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ee04d3a616dfbeb7"}},{"code_sha256_prefix":"1285d0bc9a86d9d8","entry":"refactor_optims","repo":"facebookresearch/nevergrad","repo_kind":"named_in_paper","path":"nevergrad/benchmark/experiments.py","file_url":"https://github.com/facebookresearch/nevergrad/blob/HEAD/nevergrad/benchmark/experiments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1285d0bc9a86d9d8"}},{"code_sha256_prefix":"bfc973cc77c53d46","entry":"remove_parens","repo":"facebookresearch/nevergrad","repo_kind":"named_in_paper","path":"nevergrad/benchmark/exporttable.py","file_url":"https://github.com/facebookresearch/nevergrad/blob/HEAD/nevergrad/benchmark/exporttable.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"bfc973cc77c53d46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}