{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/practical-bayesian-optimization-of-machine","title":"Practical Bayesian Optimization of Machine Learning Algorithms","arxiv_id":"1206.2944","date":"2012-06-13","proceeding":"NeurIPS 2012 12","authors":["Jasper Snoek","Hugo Larochelle","Ryan P. Adams"],"abstract":"Machine learning algorithms frequently require careful tuning of model\nhyperparameters, regularization terms, and optimization parameters.\nUnfortunately, this tuning is often a \"black art\" that requires expert\nexperience, unwritten rules of thumb, or sometimes brute-force search. Much\nmore appealing is the idea of developing automatic approaches which can\noptimize the performance of a given learning algorithm to the task at hand. In\nthis work, we consider the automatic tuning problem within the framework of\nBayesian optimization, in which a learning algorithm's generalization\nperformance is modeled as a sample from a Gaussian process (GP). The tractable\nposterior distribution induced by the GP leads to efficient use of the\ninformation gathered by previous experiments, enabling optimal choices about\nwhat parameters to try next. Here we show how the effects of the Gaussian\nprocess prior and the associated inference procedure can have a large impact on\nthe success or failure of Bayesian optimization. We show that thoughtful\nchoices can lead to results that exceed expert-level performance in tuning\nmachine learning algorithms. We also describe new algorithms that take into\naccount the variable cost (duration) of learning experiments and that can\nleverage the presence of multiple cores for parallel experimentation. We show\nthat these proposed algorithms improve on previous automatic procedures and can\nreach or surpass human expert-level optimization on a diverse set of\ncontemporary algorithms including latent Dirichlet allocation, structured SVMs\nand convolutional neural networks.","url_abs":"http://arxiv.org/abs/1206.2944v2","url_pdf":"http://arxiv.org/pdf/1206.2944v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"practical-bayesian-optimization-of-machine","repo_url":"https://github.com/Argaadya/intro-bayesian","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"practical-bayesian-optimization-of-machine","repo_url":"https://github.com/c-bata/goptuna","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"practical-bayesian-optimization-of-machine","repo_url":"https://github.com/HIPS/Spearmint","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"practical-bayesian-optimization-of-machine","repo_url":"https://github.com/JasperSnoek/spearmint","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"}],"methods":[{"method_slug":"gaussian-process","method_name":"Gaussian Process"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"GP EI","rank_in_archive_order":203,"of":265,"metrics":{"Percentage correct":"90.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1206.2944","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}