{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/massively-parallel-hyperparameter-tuning","title":"A System for Massively Parallel Hyperparameter Tuning","arxiv_id":"1810.05934","date":"2018-10-13","proceeding":"ICLR 2018 1","authors":["Liam Li","Kevin Jamieson","Afshin Rostamizadeh","Ekaterina Gonina","Moritz Hardt","Benjamin Recht","Ameet Talwalkar"],"abstract":"Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.","url_abs":"https://arxiv.org/abs/1810.05934v5","url_pdf":"https://arxiv.org/pdf/1810.05934v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"massively-parallel-hyperparameter-tuning","repo_url":"https://github.com/liamcli/darts_asha","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"massively-parallel-hyperparameter-tuning","repo_url":"https://github.com/Joshua-Ogbebor/hyper_param-optim","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"massively-parallel-hyperparameter-tuning","repo_url":"https://github.com/awslabs/syne-tune","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"massively-parallel-hyperparameter-tuning","repo_url":"https://github.com/c-bata/goptuna","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"massively-parallel-hyperparameter-tuning","repo_url":"https://github.com/lukashedegaard/ride","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"massively-parallel-hyperparameter-tuning","repo_url":"https://github.com/optuna/kurobako","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"distributed-computing","task_name":"Distributed Computing"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1810.05934","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1810.05934"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liamcli/darts_asha","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/c-bata/goptuna","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Joshua-Ogbebor/hyper_param-optim","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/optuna/kurobako","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/awslabs/syne-tune","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lukashedegaard/ride","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":15},"by_repo_kind":{"listed":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8b642ae98586389c","entry":"apply_init_args","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/core.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b642ae98586389c"}},{"code_sha256_prefix":"e2db25de373de572","entry":"cat_steps","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/lifecycle.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/lifecycle.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2db25de373de572"}},{"code_sha256_prefix":"3a77b599b00adde4","entry":"detach_to_cpu","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/lifecycle.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/lifecycle.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a77b599b00adde4"}},{"code_sha256_prefix":"c07ed198a634bd89","entry":"discounted_steps_per_epoch","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/optimizers.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/optimizers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c07ed198a634bd89"}},{"code_sha256_prefix":"1ac34a74721355dc","entry":"fig2img","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/logging.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ac34a74721355dc"}},{"code_sha256_prefix":"9170c3ab64f34ec1","entry":"get_log_dir","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/logging.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9170c3ab64f34ec1"}},{"code_sha256_prefix":"0a139178de9f1bb5","entry":"get_modules_to_unfreeze","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/unfreeze.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/unfreeze.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a139178de9f1bb5"}},{"code_sha256_prefix":"416813b50c6c11a1","entry":"linear_unfreeze_schedule","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/unfreeze.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/unfreeze.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"416813b50c6c11a1"}},{"code_sha256_prefix":"1a777ae82c0e6442","entry":"prefix_keys","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/lifecycle.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/lifecycle.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1a777ae82c0e6442"}},{"code_sha256_prefix":"0539e693642efec3","entry":"scatter_plot","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/feature_visualisation.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/feature_visualisation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0539e693642efec3"}},{"code_sha256_prefix":"684a9bda1377854e","entry":"sort_out_figures","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/metrics.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"684a9bda1377854e"}},{"code_sha256_prefix":"a3813ef3bc3516ed","entry":"topks_correct","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/metrics.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3813ef3bc3516ed"}},{"code_sha256_prefix":"6d14d68c20596b92","entry":"try_pickle_load","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/finetune.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d14d68c20596b92"}},{"code_sha256_prefix":"dc96d15db03f89da","entry":"try_pyth_load","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/finetune.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc96d15db03f89da"}},{"code_sha256_prefix":"4915f88063dd9ab4","entry":"unfreeze_from_end","repo":"lukashedegaard/ride","repo_kind":"listed","path":"ride/unfreeze.py","file_url":"https://github.com/lukashedegaard/ride/blob/HEAD/ride/unfreeze.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4915f88063dd9ab4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}