{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/random-search-and-reproducibility-for-neural","title":"Random Search and Reproducibility for Neural Architecture Search","arxiv_id":"1902.07638","date":"2019-02-20","proceeding":null,"authors":["Liam Li","Ameet Talwalkar"],"abstract":"Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks---PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.","url_abs":"https://arxiv.org/abs/1902.07638v3","url_pdf":"https://arxiv.org/pdf/1902.07638v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"random-search-and-reproducibility-for-neural","repo_url":"https://github.com/liamcli/randomNAS_release","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"random-search-and-reproducibility-for-neural","repo_url":"https://github.com/ben-arnao/StoRM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"random-search-and-reproducibility-for-neural","repo_url":"https://github.com/ben-arnao/stochasticmutatortuner","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"random-search-and-reproducibility-for-neural","repo_url":"https://github.com/microsoft/nn-meter","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"}],"methods":[{"method_slug":"random-search","method_name":"Random Search"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/neural-architecture-search-on-nas-bench-201-1","task":"Neural Architecture Search","dataset":"NAS-Bench-201, CIFAR-10","model":"RSPS","rank_in_archive_order":32,"of":37,"metrics":{"Accuracy (Test)":"87.66","Accuracy (Val)":"84.16","Search time (s)":"7587"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-nas-bench-201-2","task":"Neural Architecture Search","dataset":"NAS-Bench-201, CIFAR-100","model":"RSPS","rank_in_archive_order":33,"of":40,"metrics":{"Accuracy (Test)":"58.33","Accuracy (Val)":"59.00","Search time (s)":"7587"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-nas-bench-201","task":"Neural Architecture Search","dataset":"NAS-Bench-201, ImageNet-16-120","model":"RSPS","rank_in_archive_order":41,"of":49,"metrics":{"Accuracy (Test)":"31.14","Search time (s)":"7587"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1902.07638","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1902.07638"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liamcli/randomNAS_release","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ben-arnao/stochasticmutatortuner","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ben-arnao/StoRM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/nn-meter","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"named_in_paper":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"d3f2511e0002a6af","entry":"convert_models","repo":"microsoft/nn-meter","repo_kind":"listed","path":"nn_meter/builder/nn_meter_builder.py","file_url":"https://github.com/microsoft/nn-meter/blob/HEAD/nn_meter/builder/nn_meter_builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3f2511e0002a6af"}},{"code_sha256_prefix":"7a5022a04158a95d","entry":"load_config_file","repo":"microsoft/nn-meter","repo_kind":"listed","path":"nn_meter/builder/config_manager.py","file_url":"https://github.com/microsoft/nn-meter/blob/HEAD/nn_meter/builder/config_manager.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a5022a04158a95d"}},{"code_sha256_prefix":"c2ec14f99d80dc2b","entry":"load_predictor_config","repo":"microsoft/nn-meter","repo_kind":"listed","path":"nn_meter/predictor/nn_meter_predictor.py","file_url":"https://github.com/microsoft/nn-meter/blob/HEAD/nn_meter/predictor/nn_meter_predictor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2ec14f99d80dc2b"}},{"code_sha256_prefix":"4be4b8fdd0aa0d3a","entry":"main","repo":"liamcli/randomNAS_release","repo_kind":"named_in_paper","path":"searchers/random_weight_share.py","file_url":"https://github.com/liamcli/randomNAS_release/blob/HEAD/searchers/random_weight_share.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4be4b8fdd0aa0d3a"}},{"code_sha256_prefix":"b71a0773c6a09b13","entry":"merge_info","repo":"microsoft/nn-meter","repo_kind":"listed","path":"nn_meter/builder/utils.py","file_url":"https://github.com/microsoft/nn-meter/blob/HEAD/nn_meter/builder/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b71a0773c6a09b13"}},{"code_sha256_prefix":"c58336805ae6a6ab","entry":"onnx_model_to_graph","repo":"microsoft/nn-meter","repo_kind":"listed","path":"nn_meter/ir_converter/utils.py","file_url":"https://github.com/microsoft/nn-meter/blob/HEAD/nn_meter/ir_converter/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c58336805ae6a6ab"}},{"code_sha256_prefix":"9370e87a7f7cbebd","entry":"profile_models","repo":"microsoft/nn-meter","repo_kind":"listed","path":"nn_meter/builder/nn_meter_builder.py","file_url":"https://github.com/microsoft/nn-meter/blob/HEAD/nn_meter/builder/nn_meter_builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9370e87a7f7cbebd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}