Papers › Factor Normalization for Deep Neural Network Models
Factor Normalization for Deep Neural Network Models
Haobo Qi, Jing Zhou, Hansheng Wang
Deep neural network (DNN) models often involve features of ultrahigh dimensions. In most cases, the ultrahigh dimensional features can be decomposed into two parts. The first part is a low-dimensional factor model. The second part is the residual feature, with much-reduced variability and interfeature correlation. This leads to a number of interesting theoretical findings for deep neural network training. Accordingly, we are inspired to develop a new factor normalization method for fast deep neural network training. The proposed method contains three interesting steps. First, it decomposes the ultrahigh dimensional features into two parts. One is due to factors and the other one is due to residuals. Second, it modifies a given DNN model slightly so that the effect of the latent factor and residual feature can be processed separately. Last, to train a modified DNN model, a new SGD algorithm is developed that allows adaptive learning rates for the factor part and the residual part. A number of empirical experiments are presented to demonstrate its superior performance. The code is available at https://github.com/HazardNeo4869/FactorNormalization
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections