{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/overfitting-in-bayesian-optimization-an","title":"Automatic Termination for Hyperparameter Optimization","arxiv_id":"2104.08166","date":"2021-04-16","proceeding":null,"authors":["Anastasia Makarova","Huibin Shen","Valerio Perrone","Aaron Klein","Jean Baptiste Faddoul","Andreas Krause","Matthias Seeger","Cedric Archambeau"],"abstract":"Bayesian optimization (BO) is a widely popular approach for the hyperparameter optimization (HPO) in machine learning. At its core, BO iteratively evaluates promising configurations until a user-defined budget, such as wall-clock time or number of iterations, is exhausted. While the final performance after tuning heavily depends on the provided budget, it is hard to pre-specify an optimal value in advance. In this work, we propose an effective and intuitive termination criterion for BO that automatically stops the procedure if it is sufficiently close to the global optimum. Our key insight is that the discrepancy between the true objective (predictive performance on test data) and the computable target (validation performance) suggests stopping once the suboptimality in optimizing the target is dominated by the statistical estimation error. Across an extensive range of real-world HPO problems and baselines, we show that our termination criterion achieves a better trade-off between the test performance and optimization time. Additionally, we find that overfitting may occur in the context of HPO, which is arguably an overlooked problem in the literature, and show how our termination criterion helps to mitigate this phenomenon on both small and large datasets.","url_abs":"https://arxiv.org/abs/2104.08166v4","url_pdf":"https://arxiv.org/pdf/2104.08166v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"overfitting-in-bayesian-optimization-an","repo_url":"https://github.com/amazon-research/bo-early-stopping","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"}],"methods":[{"method_slug":"hpo","method_name":"HPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.08166","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08166"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-research/bo-early-stopping","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"14b7afc993448c17","entry":"extract_algo","repo":"amazon-research/bo-early-stopping","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/amazon-research/bo-early-stopping/blob/HEAD/src/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14b7afc993448c17"}},{"code_sha256_prefix":"99523aea225a3952","entry":"extract_metric","repo":"amazon-research/bo-early-stopping","repo_kind":"official","path":"bin/start_gap_estimation.py","file_url":"https://github.com/amazon-research/bo-early-stopping/blob/HEAD/bin/start_gap_estimation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99523aea225a3952"}},{"code_sha256_prefix":"be1a632e2b500b21","entry":"get_overfit_iter","repo":"amazon-research/bo-early-stopping","repo_kind":"official","path":"bin/gen_csv.py","file_url":"https://github.com/amazon-research/bo-early-stopping/blob/HEAD/bin/gen_csv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be1a632e2b500b21"}},{"code_sha256_prefix":"7f22afc4b599f122","entry":"get_stop_info_conv","repo":"amazon-research/bo-early-stopping","repo_kind":"official","path":"src/stop_methods.py","file_url":"https://github.com/amazon-research/bo-early-stopping/blob/HEAD/src/stop_methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f22afc4b599f122"}},{"code_sha256_prefix":"645b1912976d2844","entry":"get_stop_info_ours","repo":"amazon-research/bo-early-stopping","repo_kind":"official","path":"src/stop_methods.py","file_url":"https://github.com/amazon-research/bo-early-stopping/blob/HEAD/src/stop_methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"645b1912976d2844"}},{"code_sha256_prefix":"796b91add8ea7c9d","entry":"get_stop_info_ours_corrected","repo":"amazon-research/bo-early-stopping","repo_kind":"official","path":"src/stop_methods.py","file_url":"https://github.com/amazon-research/bo-early-stopping/blob/HEAD/src/stop_methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"796b91add8ea7c9d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}