{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continuously-differentiable-exponential","title":"Continuously Differentiable Exponential Linear Units","arxiv_id":"1704.07483","date":"2017-04-24","proceeding":null,"authors":["Jonathan T. Barron"],"abstract":"Exponential Linear Units (ELUs) are a useful rectifier for constructing deep\nlearning architectures, as they may speed up and otherwise improve learning by\nvirtue of not have vanishing gradients and by having mean activations near\nzero. However, the ELU activation as parametrized in [1] is not continuously\ndifferentiable with respect to its input when the shape parameter alpha is not\nequal to 1. We present an alternative parametrization which is C1 continuous\nfor all values of alpha, making the rectifier easier to reason about and making\nalpha easier to tune. This alternative parametrization has several other useful\nproperties that the original parametrization of ELU does not: 1) its derivative\nwith respect to x is bounded, 2) it contains both the linear transfer function\nand ReLU as special cases, and 3) it is scale-similar with respect to alpha.","url_abs":"http://arxiv.org/abs/1704.07483v1","url_pdf":"http://arxiv.org/pdf/1704.07483v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continuously-differentiable-exponential","repo_url":"https://github.com/pytorch/pytorch/blob/main/torch/nn/modules/activation.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[],"methods":[{"method_slug":"celu","method_name":"CELU"},{"method_slug":"elu","method_name":"ELU"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1704.07483","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}