{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/removing-the-feature-correlation-effect-of","title":"Removing the Feature Correlation Effect of Multiplicative Noise","arxiv_id":"1809.07023","date":"2018-09-19","proceeding":"NeurIPS 2018 12","authors":["Zijun Zhang","Yining Zhang","Zongpeng Li"],"abstract":"Multiplicative noise, including dropout, is widely used to regularize deep\nneural networks (DNNs), and is shown to be effective in a wide range of\narchitectures and tasks. From an information perspective, we consider injecting\nmultiplicative noise into a DNN as training the network to solve the task with\nnoisy information pathways, which leads to the observation that multiplicative\nnoise tends to increase the correlation between features, so as to increase the\nsignal-to-noise ratio of information pathways. However, high feature\ncorrelation is undesirable, as it increases redundancy in representations. In\nthis work, we propose non-correlating multiplicative noise (NCMN), which\nexploits batch normalization to remove the correlation effect in a simple yet\neffective way. We show that NCMN significantly improves the performance of\nstandard multiplicative noise on image classification tasks, providing a better\nalternative to dropout for batch-normalized networks. Additionally, we present\na unified view of NCMN and shake-shake regularization, which explains the\nperformance gain of the latter.","url_abs":"http://arxiv.org/abs/1809.07023v1","url_pdf":"http://arxiv.org/pdf/1809.07023v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"removing-the-feature-correlation-effect-of","repo_url":"https://github.com/zj10/NCMN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"feature-correlation","task_name":"Feature Correlation"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1809.07023","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}