{"url":"/method/modrelu","slug":"modrelu","name":"modReLU","full_name":"modReLU","full_name_withheld":false,"description_markdown":"**modReLU** is an activation that is a modification of a [ReLU](https://paperswithcode.com/method/relu). It is a pointwise nonlinearity, $\\sigma\\_{modReLU}\\left(z\\right) : C \\rightarrow C$, which affects only the absolute value of a complex number, defined as:\r\n\r\n$$ \\sigma\\_{modReLU}\\left(z\\right) = \\left(|z| + b\\right)\\frac{z}{|z|} \\text{ if } |z| + b \\geq 0 $$\r\n$$ \\sigma\\_{modReLU}\\left(z\\right) = 0 \\text{ if } |z| + b \\leq 0 $$\r\n\r\nwhere $b \\in \\mathbb{R}$ is a bias parameter of the nonlinearity. For a $n\\_{h}$ dimensional hidden space we learn $n\\_{h}$ nonlinearity bias parameters, one per dimension.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Unitary Evolution Recurrent Neural Networks","paper":"/paper/unitary-evolution-recurrent-neural-networks","first_author":"Martin Arjovsky","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/unitary-evolution-recurrent-neural-networks"},"source":{"url":"http://arxiv.org/abs/1511.06464v4","title":"Unitary Evolution Recurrent Neural Networks","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Adaptive Activation Functions","url":"/methods/category/adaptive-activation-functions","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Activation Functions","url":"/methods/category/activation-functions","pwc_aliases":[]}],"n_papers_tagged":6,"archive_num_papers":6,"papers_newest_first":[{"paper":"/paper/the-fft-strikes-back-an-efficient-alternative","title":"SPECTRE: An FFT-Based Efficient Drop-In Replacement to Self-Attention for Long Contexts","date":"2025-02-25","arxiv_id":"2502.18394","n_code_links":2,"syntology":null},{"paper":null,"title":"Optimal approximation using complex-valued neural networks","date":"2023-03-29","arxiv_id":"2303.16813","n_code_links":0,"syntology":null},{"paper":null,"title":"Quantitative approximation results for complex-valued neural networks","date":"2021-02-25","arxiv_id":"2102.13092","n_code_links":0,"syntology":null},{"paper":"/paper/an-automl-based-approach-to-multimodal-image","title":"An AutoML-based Approach to Multimodal Image Sentiment Analysis","date":"2021-02-16","arxiv_id":"2102.08092","n_code_links":0,"syntology":null},{"paper":"/paper/complex-unitary-recurrent-neural-networks","title":"Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform","date":"2018-11-09","arxiv_id":"1811.04142","n_code_links":1,"syntology":null},{"paper":"/paper/unitary-evolution-recurrent-neural-networks","title":"Unitary Evolution Recurrent Neural Networks","date":"2015-11-20","arxiv_id":"1511.06464","n_code_links":2,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}}],"papers_shown":6,"tasks":[{"task":"/task/automl","name":"AutoML","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/marketing","name":"Marketing","papers":1},{"task":"/task/multimodal-sentiment-analysis","name":"Multimodal Sentiment Analysis","papers":1},{"task":"/task/recommendation-systems","name":"Recommendation Systems","papers":1},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":1},{"task":"/task/sequential-image-classification","name":"Sequential Image Classification","papers":1}],"tasks_shown":8,"n_tasks":8,"usage_by_year":[{"year":"2015","papers":1},{"year":"2018","papers":1},{"year":"2021","papers":2},{"year":"2023","papers":1},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/modrelu"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}