Papers › Generalization Properties of NAS under Activation and Skip Connection Search
Generalization Properties of NAS under Activation and Skip Connection Search
Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Neural Architecture Search (NAS) has fostered the automatic discovery of state-of-the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generalization properties of NAS under a unifying framework enabling (deep) layer skip connection search and activation function search. To this end, we derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime using a certain search space including mixed activation functions, fully connected, and residual neural networks. We use the minimum eigenvalue to establish generalization error bounds of NAS in the stochastic gradient descent training. Importantly, we theoretically and experimentally show how the derived results can guide NAS to select the top-performing architectures, even in the case without training, leading to a train-free algorithm based on our theory. Accordingly, our numerical validation shed light on the design of computationally efficient methods for NAS. Our analysis is non-trivial due to the coupling of various architectures and activation functions under the unifying framework and has its own interest in providing the lower bound of the minimum eigenvalue of NTK in deep learning theory.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Neural Architecture Search | NATS-Bench Topology, CIFAR-10 | EigenNas (Zhu et al., 2022) | Test Accuracy | 93.46 | #6 of 11 | Archive leaderboard | report |
| Neural Architecture Search | NATS-Bench Topology, CIFAR-100 | EigenNas (Zhu et al., 2022) | Test Accuracy | 71.42 | #3 of 11 | Archive leaderboard | report |
| Neural Architecture Search | NATS-Bench Topology, ImageNet16-120 | EigenNas (Zhu et al., 2022) | Test Accuracy | 45.54 | #2 of 11 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections