{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generalizing-few-shot-nas-with-gradient-1","title":"Generalizing Few-Shot NAS with Gradient Matching","arxiv_id":"2203.15207","date":"2022-03-29","proceeding":"ICLR 2022 4","authors":["Shoukang Hu","Ruochen Wang","Lanqing Hong","Zhenguo Li","Cho-Jui Hsieh","Jiashi Feng"],"abstract":"Efficient performance estimation of architectures drawn from large search spaces is essential to Neural Architecture Search. One-Shot methods tackle this challenge by training one supernet to approximate the performance of every architecture in the search space via weight-sharing, thereby drastically reducing the search cost. However, due to coupled optimization between child architectures caused by weight-sharing, One-Shot supernet's performance estimation could be inaccurate, leading to degraded search outcomes. To address this issue, Few-Shot NAS reduces the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets via edge-wise (layer-wise) exhaustive partitioning. Since each partition of the supernet is not equally important, it necessitates the design of a more effective splitting criterion. In this work, we propose a gradient matching score (GM) that leverages gradient information at the shared weight for making informed splitting decisions. Intuitively, gradients from different child models can be used to identify whether they agree on how to update the shared modules, and subsequently to decide if they should share the same weight. Compared with exhaustive partitioning, the proposed criterion significantly reduces the branching factor per edge. This allows us to split more edges (layers) for a given budget, resulting in substantially improved performance as NAS search spaces usually include dozens of edges (layers). Extensive empirical evaluations of the proposed method on a wide range of search spaces (NASBench-201, DARTS, MobileNet Space), datasets (cifar10, cifar100, ImageNet) and search algorithms (DARTS, SNAS, RSPS, ProxylessNAS, OFA) demonstrate that it significantly outperforms its Few-Shot counterparts while surpassing previous comparable methods in terms of the accuracy of derived architectures.","url_abs":"https://arxiv.org/abs/2203.15207v2","url_pdf":"https://arxiv.org/pdf/2203.15207v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generalizing-few-shot-nas-with-gradient-1","repo_url":"https://github.com/skhu101/GM-NAS","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"architecture-search","task_name":"Neural Architecture Search"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"cutout","method_name":"Cutout"},{"method_slug":"darts","method_name":"DARTS"},{"method_slug":"droppath","method_name":"DropPath"},{"method_slug":"proxylessnas","method_name":"ProxylessNAS"},{"method_slug":"reinforce","method_name":"REINFORCE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.15207","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.15207"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/skhu101/GM-NAS","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5954e4a7e67b4a0a","entry":"checkpoint_metric","repo":"skhu101/GM-NAS","repo_kind":"official","path":"Imagenet_train/avg_checkpoints.py","file_url":"https://github.com/skhu101/GM-NAS/blob/HEAD/Imagenet_train/avg_checkpoints.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5954e4a7e67b4a0a"}},{"code_sha256_prefix":"c1806216600c618d","entry":"conv2d","repo":"skhu101/GM-NAS","repo_kind":"official","path":"Proxylessnas-GM/proxyless_nas_tensorflow/tf_layers.py","file_url":"https://github.com/skhu101/GM-NAS/blob/HEAD/Proxylessnas-GM/proxyless_nas_tensorflow/tf_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1806216600c618d"}},{"code_sha256_prefix":"b89add904c0b2340","entry":"cross_entropy_with_label_smoothing","repo":"skhu101/GM-NAS","repo_kind":"official","path":"Proxylessnas-GM/proxyless_nas/utils.py","file_url":"https://github.com/skhu101/GM-NAS/blob/HEAD/Proxylessnas-GM/proxyless_nas/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b89add904c0b2340"}},{"code_sha256_prefix":"5a736b0013a9cf38","entry":"depthwise_conv2d","repo":"skhu101/GM-NAS","repo_kind":"official","path":"Proxylessnas-GM/proxyless_nas_tensorflow/tf_layers.py","file_url":"https://github.com/skhu101/GM-NAS/blob/HEAD/Proxylessnas-GM/proxyless_nas_tensorflow/tf_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a736b0013a9cf38"}},{"code_sha256_prefix":"ecbe897d347181d9","entry":"download_url","repo":"skhu101/GM-NAS","repo_kind":"official","path":"Proxylessnas-GM/proxyless_nas/utils.py","file_url":"https://github.com/skhu101/GM-NAS/blob/HEAD/Proxylessnas-GM/proxyless_nas/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecbe897d347181d9"}},{"code_sha256_prefix":"a7f652bd280d580e","entry":"load_url","repo":"skhu101/GM-NAS","repo_kind":"official","path":"Proxylessnas-GM/proxyless_nas/utils.py","file_url":"https://github.com/skhu101/GM-NAS/blob/HEAD/Proxylessnas-GM/proxyless_nas/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7f652bd280d580e"}},{"code_sha256_prefix":"d7328b8252e400cf","entry":"proxyless_base","repo":"skhu101/GM-NAS","repo_kind":"official","path":"Proxylessnas-GM/proxyless_nas/model_zoo.py","file_url":"https://github.com/skhu101/GM-NAS/blob/HEAD/Proxylessnas-GM/proxyless_nas/model_zoo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7328b8252e400cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}