{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nats-bench-benchmarking-nas-algorithms-for","title":"NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size","arxiv_id":"2009.00437","date":"2020-08-28","proceeding":null,"authors":["Xuanyi Dong","Lu Liu","Katarzyna Musial","Bogdan Gabrys"],"abstract":"Neural architecture search (NAS) has attracted a lot of attention and has been illustrated to bring tangible benefits in a large number of applications in the past few years. Architecture topology and architecture size have been regarded as two of the most important aspects for the performance of deep learning models and the community has spawned lots of searching algorithms for both aspects of the neural architectures. However, the performance gain from these searching algorithms is achieved under different search spaces and training setups. This makes the overall performance of the algorithms to some extent incomparable and the improvement from a sub-module of the searching model unclear. In this paper, we propose NATS-Bench, a unified benchmark on searching for both topology and size, for (almost) any up-to-date NAS algorithm. NATS-Bench includes the search space of 15,625 neural cell candidates for architecture topology and 32,768 for architecture size on three datasets. We analyze the validity of our benchmark in terms of various criteria and performance comparison of all candidates in the search space. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally cost friendly environment. All codes are publicly available at: https://xuanyidong.com/assets/projects/NATS-Bench.","url_abs":"https://arxiv.org/abs/2009.00437v6","url_pdf":"https://arxiv.org/pdf/2009.00437v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nats-bench-benchmarking-nas-algorithms-for","repo_url":"https://github.com/D-X-Y/AutoDL-Projects","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"nats-bench-benchmarking-nas-algorithms-for","repo_url":"https://github.com/D-X-Y/NATS-Bench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"nats-bench-benchmarking-nas-algorithms-for","repo_url":"https://github.com/Edge-AI-Acceleration-Lab/RBFleX-NAS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"nats-bench-benchmarking-nas-algorithms-for","repo_url":"https://github.com/tomomasayamasaki/RBFleX-NAS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"}],"methods":[],"datasets_introduced":[{"slug":"nats-bench","name":"NATS-Bench","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2009.00437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.00437"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/D-X-Y/AutoDL-Projects","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/D-X-Y/NATS-Bench","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tomomasayamasaki/RBFleX-NAS","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Edge-AI-Acceleration-Lab/RBFleX-NAS","reach":null}],"summary":{"ran_draft_wrong":4},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"8cc3e754d12db97c","entry":"Simularity_Mat","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8cc3e754d12db97c"}},{"code_sha256_prefix":"de4d734af0f0f0c4","entry":"normalize","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"de4d734af0f0f0c4"}},{"code_sha256_prefix":"842a6c1a4fd06ffd","entry":"print_information","repo":"D-X-Y/NATS-Bench","repo_kind":"listed","path":"nats_bench/api_size.py","file_url":"https://github.com/D-X-Y/NATS-Bench/blob/HEAD/nats_bench/api_size.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"842a6c1a4fd06ffd"}},{"code_sha256_prefix":"0704483788b12d8b","entry":"print_information","repo":"D-X-Y/NATS-Bench","repo_kind":"listed","path":"nats_bench/api_topology.py","file_url":"https://github.com/D-X-Y/NATS-Bench/blob/HEAD/nats_bench/api_topology.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0704483788b12d8b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}