Papers › How does topology of neural architectures impact gradient propagation and model performance?

How does topology of neural architectures impact gradient propagation and model performance?

16 Jun 2021CVPR 2021 6archive 2025-07-28

Kartikeya Bhardwa, Guihong Li2, Radu Marculescu

DenseNets introduce concatenation-type skip connections that achieve state-of-the-art accuracy in several computer vision tasks. In this paper, we reveal that the topology of the concatenation-type skip connections is closely related to the gradient propagation which, in turn, enables a predictable behavior of DNNs’ test performance. To this end, we introduce a new metric called NN-Mass to quantify how effectively information flows through DNNs. Moreover, we empirically show that NN-Mass also works for other types of skip connections, e.g., for ResNets, Wide-ResNets (WRNs), and MobileNets, which contain addition-type skip connections (i.e., residuals or inverted residuals). As such, for both DenseNet-like CNNs and ResNets/WRNs/MobileNets, our theoretically grounded NN-Mass can identify models with similar accuracy, despite having significantly different size/compute requirements. Detailed experiments on both synthetic and real datasets (e.g., MNIST, CIFAR-10, CIFAR100, ImageNet) provide extensive evidence for our insights. Finally, the closed-form equation of our NN-Mass enables us to design significantly compressed DenseNets (for CIFAR10) and MobileNets (for ImageNet) directly at initialization without time-consuming training and/or searching.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

SLDGroup/NN_Mass officialmentioned in paperpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to How does topology influence gradient propagation and model performance of deep... (arXiv:1910.00780; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Model CompressionNeural Architecture Search

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A FLOPS 1.95G #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A Parameters 5.02M #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A Search Time (GPU days) 0 #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-A Top-1 Error Rate 3.0% #35 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C FLOPS 1.2G #36 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C Parameters 3.82M #36 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C Search Time (GPU days) 0 #36 of 41 Archive leaderboard report
Neural Architecture Search CIFAR-10 NN-MASS- CIFAR-C Top-1 Error Rate 3.18% #36 of 41 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B Accuracy 73.3 #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B FLOPs 393M #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B MACs 393M #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B Params 3.7M #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-B Top-1 Error Rate 26.7 #123 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A Accuracy 72.9 #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A FLOPs 200M #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A MACs 200M #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A Params 2.3M #126 of 135 Archive leaderboard report
Neural Architecture Search ImageNet NN-MASS-A Top-1 Error Rate 27.1 #126 of 135 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections