{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/randomout-using-a-convolutional-gradient-norm","title":"RandomOut: Using a convolutional gradient norm to rescue convolutional filters","arxiv_id":"1602.05931","date":"2016-02-18","proceeding":null,"authors":["Joseph Paul Cohen","Henry Z. Lo","Wei Ding"],"abstract":"Filters in convolutional neural networks are sensitive to their\ninitialization. The random numbers used to initialize filters are a bias and\ndetermine if you will \"win\" and converge to a satisfactory local minimum so we\ncall this The Filter Lottery. We observe that the 28x28 Inception-V3 model\nwithout Batch Normalization fails to train 26% of the time when varying the\nrandom seed alone. This is a problem that affects the trial and error process\nof designing a network. Because random seeds have a large impact it makes it\nhard to evaluate a network design without trying many different random starting\nweights. This work aims to reduce the bias imposed by the initial weights so a\nnetwork converges more consistently. We propose to evaluate and replace\nspecific convolutional filters that have little impact on the prediction. We\nuse the gradient norm to evaluate the impact of a filter on error, and\nre-initialize filters when the gradient norm of its weights falls below a\nspecific threshold. This consistently improves accuracy on the 28x28\nInception-V3 with a median increase of +3.3%. In effect our method RandomOut\nincreases the number of filters explored without increasing the size of the\nnetwork. We observe that the RandomOut method has more consistent\ngeneralization performance, having a standard deviation of 1.3% instead of 2%\nwhen varying random seeds, and does so faster and with fewer parameters.","url_abs":"http://arxiv.org/abs/1602.05931v3","url_pdf":"http://arxiv.org/pdf/1602.05931v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"randomout-using-a-convolutional-gradient-norm","repo_url":"https://github.com/ieee8023/NeuralNetwork-Examples","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"mxnet","reach":null}],"tasks":[],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"auxiliary-classifier","method_name":"Auxiliary Classifier"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"inception-v3","method_name":"Inception-v3"},{"method_slug":"inception-v3-module","method_name":"Inception-v3 Module"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}