{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-importance-of-normalisation-layers-in","title":"On the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation Units","arxiv_id":"1508.00330","date":"2015-08-03","proceeding":null,"authors":["Zhibin Liao","Gustavo Carneiro"],"abstract":"Deep feedforward neural networks with piecewise linear activations are\ncurrently producing the state-of-the-art results in several public datasets.\nThe combination of deep learning models and piecewise linear activation\nfunctions allows for the estimation of exponentially complex functions with the\nuse of a large number of subnetworks specialized in the classification of\nsimilar input examples. During the training process, these subnetworks avoid\noverfitting with an implicit regularization scheme based on the fact that they\nmust share their parameters with other subnetworks. Using this framework, we\nhave made an empirical observation that can improve even more the performance\nof such models. We notice that these models assume a balanced initial\ndistribution of data points with respect to the domain of the piecewise linear\nactivation function. If that assumption is violated, then the piecewise linear\nactivation units can degenerate into purely linear activation units, which can\nresult in a significant reduction of their capacity to learn complex functions.\nFurthermore, as the number of model layers increases, this unbalanced initial\ndistribution makes the model ill-conditioned. Therefore, we propose the\nintroduction of batch normalisation units into deep feedforward neural networks\nwith piecewise linear activations, which drives a more balanced use of these\nactivation units, where each region of the activation function is trained with\na relatively large proportion of training samples. Also, this batch\nnormalisation promotes the pre-conditioning of very deep learning models. We\nshow that by introducing maxout and batch normalisation units to the network in\nnetwork model results in a model that produces classification results that are\nbetter than or comparable to the current state of the art in CIFAR-10,\nCIFAR-100, MNIST, and SVHN datasets.","url_abs":"http://arxiv.org/abs/1508.00330v2","url_pdf":"http://arxiv.org/pdf/1508.00330v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"maxout","method_name":"Maxout"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"MIM","rank_in_archive_order":191,"of":265,"metrics":{"Percentage correct":"91.5"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"MIM","rank_in_archive_order":174,"of":211,"metrics":{"Percentage correct":"70.8"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-mnist","task":"Image Classification","dataset":"MNIST","model":"MIM","rank_in_archive_order":29,"of":81,"metrics":{"Percentage error":"0.4"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-svhn","task":"Image Classification","dataset":"SVHN","model":"MIM","rank_in_archive_order":27,"of":62,"metrics":{"Percentage error":"2.0"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}