Papers › Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm
Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm
Meena Jagadeesan, Ilya Razenshteyn, Suriya Gunasekar
We provide a function space characterization of the inductive bias resulting from minimizing the ℓ₂ norm of the weights in multi-channel convolutional neural networks with linear activations and empirically test our resulting hypothesis on ReLU networks trained using gradient descent. We define an induced regularizer in the function space as the minimum ℓ₂ norm of weights of a network required to realize a function. For two layer linear convolutional networks with C output channels and kernel size K, we show the following: (a) If the inputs to the network are single channeled, the induced regularizer for any K is independent of the number of output channels C. Furthermore, we derive the regularizer is a norm given by a semidefinite program (SDP). (b) In contrast, for multi-channel inputs, multiple output channels can be necessary to merely realize all matrix-valued linear functions and thus the inductive bias does depend on C. However, for sufficiently large C, the induced regularizer is again given by an SDP that is independent of C. In particular, the induced regularizer for K=1 and K=D (input dimension) is given in closed form as the nuclear norm and the ℓ_(2,1) group-sparse norm, respectively, of the Fourier coefficients of the linear predictor. We investigate the broader applicability of our theoretical results to implicit regularization from gradient descent on linear and ReLU networks through experiments on MNIST and CIFAR-10 datasets.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections