{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-it-time-to-swish-comparing-deep-learning","title":"Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks","arxiv_id":"1901.02671","date":"2019-01-09","proceeding":"EMNLP 2018 10","authors":["Steffen Eger","Paul Youssef","Iryna Gurevych"],"abstract":"Activation functions play a crucial role in neural networks because they are\nthe nonlinearities which have been attributed to the success story of deep\nlearning. One of the currently most popular activation functions is ReLU, but\nseveral competitors have recently been proposed or 'discovered', including\nLReLU functions and swish. While most works compare newly proposed activation\nfunctions on few tasks (usually from image classification) and against few\ncompetitors (usually ReLU), we perform the first large-scale comparison of 21\nactivation functions across eight different NLP tasks. We find that a largely\nunknown activation function performs most stably across all tasks, the\nso-called penalized tanh function. We also show that it can successfully\nreplace the sigmoid and tanh gates in LSTM cells, leading to a 2 percentage\npoint (pp) improvement over the standard choices on a challenging NLP task.","url_abs":"http://arxiv.org/abs/1901.02671v1","url_pdf":"http://arxiv.org/pdf/1901.02671v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-it-time-to-swish-comparing-deep-learning","repo_url":"https://github.com/UKPLab/emnlp2018-activation-functions","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}