</> Code with Papers by Syntology

Papers › How does topology of neural architectures impact gradient propagation and model performance?

How does topology of neural architectures impact gradient propagation and model performance?

16 Jun 2021CVPR 2021 6archive 2025-07-28

Kartikeya Bhardwa, Guihong Li2, Radu Marculescu

DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs’ test performance. To this end, we introduce a new metric called NN-Mass to quantify how effectively information flows through DNNs. Moreover, we empirically show that NN-Mass also works for other types of skip connections, e.g., for ResNets, Wide-ResNets (WRNs), and MobileNets, which contain addition-type skip connections (i.e., residuals or inverted residuals). As such, for both DenseNet-like CNNs and ResNets/WRNs/MobileNets, our theoretically grounded NN-Mass can identify models with similar accuracy, despite having significantly different size/compute requirements. Detailed experiments on both synthetic and real datasets (e.g., MNIST, CIFAR-10, CIFAR100, ImageNet) provide extensive evidence for our insights. Finally, the closed-form equation of our NN-Mass enables us to design significantly compressed DenseNets (for CIFAR10) and MobileNets (for ImageNet) directly at initialization without time-consuming training and/or searching.

PaperPDFCode

Code

SLDGroup/NN_Mass officialmentioned in paperpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Model CompressionNeural Architecture Search

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A FLOPS 1.95G #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A Parameters 5.02M #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A Search Time (GPU days) 0 #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A Top-1 Error Rate 3.0% #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C FLOPS 1.2G #36 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C Parameters 3.82M #36 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C Search Time (GPU days) 0 #36 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C Top-1 Error Rate 3.18% #36 of 41 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B Accuracy 73.3 #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B FLOPs 393M #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B MACs 393M #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B Params 3.7M #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B Top-1 Error Rate 26.7 #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A Accuracy 72.9 #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A FLOPs 200M #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A MACs 200M #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A Params 2.3M #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A Top-1 Error Rate 27.1 #126 of 135 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections