{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/effectiveness-of-scaled-exponentially","title":"Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)","arxiv_id":"1807.10117","date":"2018-07-26","proceeding":null,"authors":["G. Zhang","H. Li"],"abstract":"Recently, self-normalizing neural networks (SNNs) have been proposed with the\nintention to avoid batch or weight normalization. The key step in SNNs is to\nproperly scale the exponential linear unit (referred to as SELU) to inherently\nincorporate normalization based on central limit theory. SELU is a\nmonotonically increasing function, where it has an approximately constant\nnegative output for large negative input. In this work, we propose a new\nactivation function to break the monotonicity property of SELU while still\npreserving the self-normalizing property. Differently from SELU, the new\nfunction introduces a bump-shaped function in the region of negative input by\nregularizing a linear function with a scaled exponential function, which is\nreferred to as a scaled exponentially-regularized linear unit (SERLU). The\nbump-shaped function has approximately zero response to large negative input\nwhile being able to push the output of SERLU towards zero mean statistically.\nTo effectively combat over-fitting, we develop a so-called shift-dropout for\nSERLU, which includes standard dropout as a special case. Experimental results\non MNIST, CIFAR10 and CIFAR100 show that SERLU-based neural networks provide\nconsistently promising results in comparison to other 5 activation functions\nincluding ELU, SELU, Swish, Leakly ReLU and ReLU.","url_abs":"http://arxiv.org/abs/1807.10117v2","url_pdf":"http://arxiv.org/pdf/1807.10117v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"elu","method_name":"ELU"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"serlu","method_name":"SERLU"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"}],"datasets_introduced":[],"methods_introduced":[{"slug":"serlu","name":"SERLU","full_name":"SERLU"}],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}