{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/projection-based-weight-normalization-for","title":"Projection Based Weight Normalization for Deep Neural Networks","arxiv_id":"1710.02338","date":"2017-10-06","proceeding":null,"authors":["Lei Huang","Xianglong Liu","Bo Lang","Bo Li"],"abstract":"Optimizing deep neural networks (DNNs) often suffers from the ill-conditioned\nproblem. We observe that the scaling-based weight space symmetry property in\nrectified nonlinear network will cause this negative effect. Therefore, we\npropose to constrain the incoming weights of each neuron to be unit-norm, which\nis formulated as an optimization problem over Oblique manifold. A simple yet\nefficient method referred to as projection based weight normalization (PBWN) is\nalso developed to solve this problem. PBWN executes standard gradient updates,\nfollowed by projecting the updated weight back to Oblique manifold. This\nproposed method has the property of regularization and collaborates well with\nthe commonly used batch normalization technique. We conduct comprehensive\nexperiments on several widely-used image datasets including CIFAR-10,\nCIFAR-100, SVHN and ImageNet for supervised learning over the state-of-the-art\nconvolutional neural networks, such as Inception, VGG and residual networks.\nThe results show that our method is able to improve the performance of DNNs\nwith different architectures consistently. We also apply our method to Ladder\nnetwork for semi-supervised learning on permutation invariant MNIST dataset,\nand our method outperforms the state-of-the-art methods: we obtain test errors\nas 2.52%, 1.06%, and 0.91% with only 20, 50, and 100 labeled samples,\nrespectively.","url_abs":"http://arxiv.org/abs/1710.02338v1","url_pdf":"http://arxiv.org/pdf/1710.02338v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"projection-based-weight-normalization-for","repo_url":"https://github.com/huangleiBuaa/NormProjection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"torch","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-normalization","method_name":"Weight Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1710.02338","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}