{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-hyperparameter-optimization-of-deep","title":"Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates","arxiv_id":"1607.08316","date":"2016-07-28","proceeding":null,"authors":["Ilija Ilievski","Taimoor Akhtar","Jiashi Feng","Christine Annette Shoemaker"],"abstract":"Automatically searching for optimal hyperparameter configurations is of\ncrucial importance for applying deep learning algorithms in practice. Recently,\nBayesian optimization has been proposed for optimizing hyperparameters of\nvarious machine learning algorithms. Those methods adopt probabilistic\nsurrogate models like Gaussian processes to approximate and minimize the\nvalidation error function of hyperparameter values. However, probabilistic\nsurrogates require accurate estimates of sufficient statistics (e.g.,\ncovariance) of the error distribution and thus need many function evaluations\nwith a sizeable number of hyperparameters. This makes them inefficient for\noptimizing hyperparameters of deep learning algorithms, which are highly\nexpensive to evaluate. In this work, we propose a new deterministic and\nefficient hyperparameter optimization method that employs radial basis\nfunctions as error surrogates. The proposed mixed integer algorithm, called\nHORD, searches the surrogate for the most promising hyperparameter values\nthrough dynamic coordinate search and requires many fewer function evaluations.\nHORD does well in low dimensions but it is exceptionally better in higher\ndimensions. Extensive evaluations on MNIST and CIFAR-10 for four deep neural\nnetworks demonstrate HORD significantly outperforms the well-established\nBayesian optimization methods such as GP, SMAC, and TPE. For instance, on\naverage, HORD is more than 6 times faster than GP-EI in obtaining the best\nconfiguration of 19 hyperparameters.","url_abs":"http://arxiv.org/abs/1607.08316v2","url_pdf":"http://arxiv.org/pdf/1607.08316v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-hyperparameter-optimization-of-deep","repo_url":"https://github.com/jekyllstein/HORDOpt.jl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"bayesian-optimization","task_name":"Bayesian Optimization"},{"task_slug":"gaussian-processes","task_name":"Gaussian Processes"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"smac","task_name":"SMAC"},{"task_slug":"smac-1","task_name":"SMAC+"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}