{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/flatten-t-swish-a-thresholded-relu-swish-like","title":"Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning","arxiv_id":"1812.06247","date":"2018-12-15","proceeding":null,"authors":["Hock Hung Chieng","Noorhaniza Wahid","Pauline Ong","Sai Raj Kishore Perla"],"abstract":"Activation functions are essential for deep learning methods to learn and\nperform complex tasks such as image classification. Rectified Linear Unit\n(ReLU) has been widely used and become the default activation function across\nthe deep learning community since 2012. Although ReLU has been popular,\nhowever, the hard zero property of the ReLU has heavily hindered the negative\nvalues from propagating through the network. Consequently, the deep neural\nnetwork has not been benefited from the negative representations. In this work,\nan activation function called Flatten-T Swish (FTS) that leverage the benefit\nof the negative values is proposed. To verify its performance, this study\nevaluates FTS with ReLU and several recent activation functions. Each\nactivation function is trained using MNIST dataset on five different deep fully\nconnected neural networks (DFNNs) with depth vary from five to eight layers.\nFor a fair evaluation, all DFNNs are using the same configuration settings.\nBased on the experimental results, FTS with a threshold value, T=-0.20 has the\nbest overall performance. As compared with ReLU, FTS (T=-0.20) improves MNIST\nclassification accuracy by 0.13%, 0.70%, 0.67%, 1.07% and 1.15% on wider 5\nlayers, slimmer 5 layers, 6 layers, 7 layers and 8 layers DFNNs respectively.\nApart from this, the study also noticed that FTS converges twice as fast as\nReLU. Although there are other existing activation functions are also\nevaluated, this study elects ReLU as the baseline activation function.","url_abs":"http://arxiv.org/abs/1812.06247v1","url_pdf":"http://arxiv.org/pdf/1812.06247v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"flatten-t-swish-a-thresholded-relu-swish-like","repo_url":"https://github.com/lessw2020/FTSwish","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"flatten-t-swish-a-thresholded-relu-swish-like","repo_url":"https://github.com/lessw2020/FTSwishPlus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}