Hyperparameter Optimization

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

1 leaderboard table shown for this task, 1 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

3 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 339 papers with code (813 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
  21 Mar 2016 17 repositories listed Syntology ran 1 of 2 samples · 1 unverified
  Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters.
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
  12 Dec 2010 15 repositories listed Syntology ran 0 of 16 samples · 16 unverified
  We present a tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions.
• Optuna: A Next-generation Hyperparameter Optimization Framework
  25 Jul 2019 11 repositories listed Syntology ran 0 of 20 samples · 20 unverified
  We will present the design-techniques that became necessary in the development of the software that meets the above criteria, and demonstrate the power of our new design through experimental results and real world…
• Optimizing Millions of Hyperparameters by Implicit Differentiation
  6 Nov 2019 9 repositories listed Syntology ran 3 of 4 samples · 1 unverified
  We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations.
• Are GANs Created Equal? A Large-Scale Study
  28 Nov 2017 9 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 2 pointer-only (licence)
  Generative adversarial networks (GAN) are a powerful subclass of generative models.
• A Tutorial on Bayesian Optimization
  8 Jul 2018 7 repositories listed Syntology ran 2 of 10 samples · 8 unverified
  It builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this…
• A System for Massively Parallel Hyperparameter Tuning
  13 Oct 2018 6 repositories listed Syntology ran 0 of 15 samples · 15 unverified
  Modern learning models are characterized by large hyperparameter spaces and long training times.
• Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks
  21 Jul 2017 6 repositories listed
  Selecting optimal parameters for a neural network architecture can often make the difference between mediocre and state-of-the-art performance.
• GPT Takes the Bar Exam
  29 Dec 2022 5 repositories listed
  Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as "the Bar Exam," as a precondition for law practice.
• MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements
  5 Dec 2020 5 repositories listed
  Instead of sampling configurations randomly in HB, BOHB samples configurations based on a BO surrogate model, which is constructed with the high-fidelity measurements only.
• Single Headed Attention RNN: Stop Thinking With Your Head
  26 Nov 2019 5 repositories listed
  The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street.
• Nystrom Method for Accurate and Scalable Implicit Differentiation
  20 Feb 2023 4 repositories listed
  The essential difficulty of gradient-based bilevel optimization using implicit differentiation is to estimate the inverse Hessian vector product with respect to neural network parameters.
• Random Search and Reproducibility for Neural Architecture Search
  20 Feb 2019 4 repositories listed Syntology ran 0 of 7 samples · 7 unverified · 1 pointer-only (licence)
  Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures.
• Benchmarking Automatic Machine Learning Frameworks
  17 Aug 2018 4 repositories listed
  AutoML serves as the bridge between varying levels of expertise when designing machine learning systems and expedites the data science process.
• Tune: A Research Platform for Distributed Model Selection and Training
  13 Jul 2018 4 repositories listed Syntology ran 0 of 9 samples · 9 unverified · 9 pointer-only (licence)
  We show that this interface meets the requirements for a broad range of hyperparameter search algorithms, allows straightforward scaling of search to large clusters, and simplifies algorithm implementation.
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale
  4 Jul 2018 4 repositories listed
  Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally…
• Scalable Bayesian Optimization Using Deep Neural Networks
  19 Feb 2015 4 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Bayesian optimization is an effective methodology for the global optimization of functions with expensive evaluations.
• Practical Bayesian Optimization of Machine Learning Algorithms
  13 Jun 2012 4 repositories listed
  In this work, we consider the automatic tuning problem within the framework of Bayesian optimization, in which a learning algorithm's generalization performance is modeled as a sample from a Gaussian process (GP).
• LEMUR Neural Network Dataset: Towards Seamless AutoML
  14 Apr 2025 3 repositories listed
  We introduce LEMUR, an open source dataset of neural network models with well-structured code for diverse architectures across tasks such as object detection, image classification, segmentation, and natural language…
• Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features
  30 Jun 2024 3 repositories listed
  Inspired by Bayesian ideas from GPR, this paper introduces a random objective function that is tailored for hyperparameter tuning of vector-valued random features.
• TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications
  6 Nov 2023 3 repositories listed Syntology ran 8 of 8 samples · 0 unverified
  We introduce TabRepo, a new dataset of tabular model evaluations and predictions.
• Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference
  8 Mar 2023 3 repositories listed Syntology ran 2 of 9 samples · 7 unverified · 9 pointer-only (licence)
  Large Language Models (LLMs) have sparked significant interest in their generative capabilities, leading to the development of various commercial applications.
• FedNest: Federated Bilevel, Minimax, and Compositional Optimization
  4 May 2022 3 repositories listed Syntology ran 2 of 4 samples · 2 unverified · 2 pointer-only (licence)
  Standard federated optimization methods successfully apply to stochastic problems with single-level structure.
• HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation
  7 Dec 2020 3 repositories listed
  Our results on the Bayesmark benchmark indicate that heteroscedasticity and non-stationarity pose significant challenges for black-box optimisers.
• Model-based Asynchronous Hyperparameter and Neural Architecture Search
  24 Mar 2020 3 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  We introduce a model-based asynchronous multi-fidelity method for hyperparameter and neural architecture search that combines the strengths of asynchronous Hyperband and Gaussian process-based Bayesian optimization.
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
  25 Oct 2019 3 repositories listed Syntology ran 0 of 8 samples · 8 unverified
  Bayesian optimization (BO), which has long had success in hyperparameter optimization, has recently emerged as a very promising strategy for NAS when it is coupled with a neural predictor.
• Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
  7 Mar 2019 3 repositories listed
  Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems.
• Automatic Gradient Boosting
  10 Jul 2018 3 repositories listed
  Automatic machine learning performs predictive modeling with high performing machine learning tools without human interference.
• Online Learning Rate Adaptation with Hypergradient Descent
  14 Mar 2017 3 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice.
• Global optimization of Lipschitz functions
  7 Mar 2017 3 repositories listed Syntology ran 1 of 3 samples · 2 unverified
  The goal of the paper is to design sequential strategies which lead to efficient optimization of an unknown function under the only assumption that it has a finite Lipschitz constant.

Syntology lines on 17 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.