{"url":"/method/srelu","slug":"srelu","name":"SReLU","full_name":"S-shaped ReLU","full_name_withheld":false,"description_markdown":"The **S-shaped Rectified Linear Unit**, or **SReLU**, is an activation function for neural networks. It learns both convex and non-convex functions, imitating the multiple function forms given by the two fundamental laws, namely  the Webner-Fechner law and the Stevens law, in psychophysics and neural sciences. Specifically, SReLU consists of three piecewise linear functions, which are formulated by four learnable parameters. \r\n\r\nThe SReLU is defined as a mapping:\r\n\r\n$$ f\\left(x\\right) = t\\_{i}^{r}  + a^{r}\\_{i}\\left(x\\_{i}-t^{r}\\_{i}\\right) \\text{ if } x\\_{i} \\geq t^{r}\\_{i} $$\r\n$$ f\\left(x\\right) = x\\_{i} \\text{ if } t^{r}\\_{i} > x > t\\_{i}^{l}$$\r\n$$ f\\left(x\\right) = t\\_{i}^{l}  + a^{l}\\_{i}\\left(x\\_{i}-t^{l}\\_{i}\\right) \\text{ if } x\\_{i} \\leq t^{l}\\_{i} $$\r\n\r\nwhere $t^{l}\\_{i}$, $t^{r}\\_{i}$ and $a^{l}\\_{i}$ are learnable parameters of the network $i$ and indicates that the SReLU can differ in different channels. The parameter $a^{r}\\_{i}$ represents the slope of the right line with input above a set threshold. $t^{r}\\_{i}$ and $t^{l}\\_{i}$ are thresholds in positive and negative directions respectively.\r\n\r\nSource: [Activation Functions](https://arxiv.org/pdf/1811.03378.pdf)","description_state":"present","introduced_year":null,"introduced_by":{"title":"Deep Learning with S-shaped Rectified Linear Activation Units","paper":"/paper/deep-learning-with-s-shaped-rectified-linear","first_author":"Xiaojie Jin","n_authors":6,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/deep-learning-with-s-shaped-rectified-linear"},"source":{"url":"http://arxiv.org/abs/1512.07030v1","title":"Deep Learning with S-shaped Rectified Linear Activation Units","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/ThomasHagebols/sReLU/blob/2180d2334dd2a39847e6bafe1033f80a655c5e07/srelu.py#L9","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Activation Functions","url":"/methods/category/activation-functions","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":null,"title":"Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units","date":"2020-06-04","arxiv_id":"2006.02797","n_code_links":0,"syntology":null},{"paper":"/paper/deep-learning-with-s-shaped-rectified-linear","title":"Deep Learning with S-shaped Rectified Linear Activation Units","date":"2015-12-22","arxiv_id":"1512.07030","n_code_links":1,"syntology":null}],"papers_shown":2,"tasks":[{"task":"/task/deep-learning","name":"Deep Learning","papers":1}],"tasks_shown":1,"n_tasks":1,"usage_by_year":[{"year":"2015","papers":1},{"year":"2020","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/srelu"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}