{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/idarts-improving-darts-by-node-normalization","title":"iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization","arxiv_id":"2108.11014","date":"2021-08-25","proceeding":null,"authors":["Huiqun Wang","Ruijie Yang","Di Huang","Yunhong Wang"],"abstract":"Differentiable ARchiTecture Search (DARTS) uses a continuous relaxation of network representation and dramatically accelerates Neural Architecture Search (NAS) by almost thousands of times in GPU-day. However, the searching process of DARTS is unstable, which suffers severe degradation when training epochs become large, thus limiting its application. In this paper, we claim that this degradation issue is caused by the imbalanced norms between different nodes and the highly correlated outputs from various operations. We then propose an improved version of DARTS, namely iDARTS, to deal with the two problems. In the training phase, it introduces node normalization to maintain the norm balance. In the discretization phase, the continuous architecture is approximated based on the similarity between the outputs of the node and the decorrelated operations rather than the values of the architecture parameters. Extensive evaluation is conducted on CIFAR-10 and ImageNet, and the error rates of 2.25\\% and 24.7\\% are reported within 0.2 and 1.9 GPU-day for architecture search respectively, which shows its effectiveness. Additional analysis also reveals that iDARTS has the advantage in robustness and generalization over other DARTS-based counterparts.","url_abs":"https://arxiv.org/abs/2108.11014v1","url_pdf":"https://arxiv.org/pdf/2108.11014v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"}],"methods":[{"method_slug":"darts","method_name":"DARTS"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/neural-architecture-search-on-cifar-10","task":"Neural Architecture Search","dataset":"CIFAR-10","model":"iDARTS +ME","rank_in_archive_order":9,"of":41,"metrics":{"Parameters":"3.6M","Search Time (GPU days)":"0.4","Top-1 Error Rate":"2.25%"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"iDARTS (ImageNet)","rank_in_archive_order":112,"of":135,"metrics":{"MACs":"568M","Params":"5.1M","Top-1 Error Rate":"24.7"},"uses_additional_data":false},{"leaderboard":"/sota/neural-architecture-search-on-imagenet","task":"Neural Architecture Search","dataset":"ImageNet","model":"iDARTS (CIFAR-10)","rank_in_archive_order":117,"of":135,"metrics":{"MACs":"578M","Params":"5.1M","Top-1 Error Rate":"25.2"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}