{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/regularizing-cnns-with-locally-constrained","title":"Regularizing CNNs with Locally Constrained Decorrelations","arxiv_id":"1611.01967","date":"2016-11-07","proceeding":null,"authors":["Pau Rodríguez","Jordi Gonzàlez","Guillem Cucurull","Josep M. Gonfaus","Xavier Roca"],"abstract":"Regularization is key for deep learning since it allows training more complex\nmodels while keeping lower levels of overfitting. However, the most prevalent\nregularizations do not leverage all the capacity of the models since they rely\non reducing the effective number of parameters. Feature decorrelation is an\nalternative for using the full capacity of the models but the overfitting\nreduction margins are too narrow given the overhead it introduces. In this\npaper, we show that regularizing negatively correlated features is an obstacle\nfor effective decorrelation and present OrthoReg, a novel regularization\ntechnique that locally enforces feature orthogonality. As a result, imposing\nlocality constraints in feature decorrelation removes interferences between\nnegatively correlated feature weights, allowing the regularizer to reach higher\ndecorrelation bounds, and reducing the overfitting more effectively. In\nparticular, we show that the models regularized with OrthoReg have higher\naccuracy bounds even when batch normalization and dropout are present.\nMoreover, since our regularization is directly performed on the weights, it is\nespecially suitable for fully convolutional neural networks, where the weight\nspace is constant compared to the feature map space. As a result, we are able\nto reduce the overfitting of state-of-the-art CNNs on CIFAR-10, CIFAR-100, and\nSVHN.","url_abs":"http://arxiv.org/abs/1611.01967v2","url_pdf":"http://arxiv.org/pdf/1611.01967v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"regularizing-cnns-with-locally-constrained","repo_url":"https://github.com/prlz77/orthoreg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"torch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1611.01967","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}