{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mathnas-if-blocks-have-a-role-in-mathematical","title":"MathNAS: If Blocks Have a Role in Mathematical Architecture Design","arxiv_id":null,"date":"2023-09-21","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Neural Architecture Search (NAS) has emerged as a favoured method for unearthing effective neural architectures. \nRecent development of large models has intensified the demand for faster search speeds and more accurate search results. \nHowever, designing large models by NAS is challenging due to the dramatical increase of search space and the associated huge performance evaluation cost. \nConsider a typical modular search space widely used in NAS, in which a neural architecture consists of $m$ block nodes and a block node has $n$ alternative blocks. \nFacing the space containing $n^m$ candidate networks, existing NAS methods attempt to find the best one by searching and evaluating candidate networks directly.\nDifferent from the general strategy that takes architecture search as a whole problem, we propose a novel divide-and-conquer strategy by making use of the modular nature of the search space.\nHere, we introduce MathNAS, a general NAS framework based on mathematical programming.  \nIn MathNAS, the performances of all possible building blocks in the search space are calculated first, and then the performance of a network is directly predicted based on the performances of its building blocks.\nAlthough estimating block performances involves network training, just as what happens for network performance evaluation in existing NAS methods, predicting network performance is completely training-free and thus extremely fast. In contrast to the $n^m$ candidate networks to evaluate in existing NAS methods, which requires training and a formidable computational burden, there are only $m*n$ possible blocks to handle in MathNAS.\nTherefore, our approach effectively reduces the complexity of network performance evaluation. \nThe superiority of MathNAS is validated on multiple large-scale CV and NLP benchmark datasets. \nNotably on ImageNet-1k, MathNAS achieves 82.5\\% top-1 accuracy, 1.2\\% and 0.96\\% higher than Swin-T and LeViT-256, respectively. \nIn addition, when deployed on mobile device, MathNAS achieves real-time search and dynamic network switching within 1s (0.4s on TX2 GPU), surpassing baseline dynamic networks in on-device performance.","url_abs":"https://openreview.net/forum?id=e1l4ZYprQH","url_pdf":"https://openreview.net/pdf?id=e1l4ZYprQH","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mathnas-if-blocks-have-a-role-in-mathematical","repo_url":"https://github.com/wangqinsi1/mathnas","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}