{"url":"/method/reglu","slug":"reglu","name":"ReGLU","full_name":"ReGLU","full_name_withheld":false,"description_markdown":"**ReGLU** is an activation function which is a variant of [GLU](https://paperswithcode.com/method/glu). The definition is as follows:\r\n\r\n$$ \\text{ReGLU}\\left(x, W, V, b, c\\right) = \\max\\left(0, xW + b\\right) \\otimes \\left(xV + c\\right) $$","description_state":"present","introduced_year":null,"introduced_by":{"title":"GLU Variants Improve Transformer","paper":"/paper/glu-variants-improve-transformer","first_author":"Noam Shazeer","n_authors":1,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/glu-variants-improve-transformer"},"source":{"url":"https://arxiv.org/abs/2002.05202v1","title":"GLU Variants Improve Transformer","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Activation Functions","url":"/methods/category/activation-functions","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":null,"title":"ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs","date":"2024-02-06","arxiv_id":"2402.03804","n_code_links":0,"syntology":null},{"paper":"/paper/glu-variants-improve-transformer","title":"GLU Variants Improve Transformer","date":"2020-02-12","arxiv_id":"2002.05202","n_code_links":27,"syntology":{"ran":9,"of":9,"unverified":0,"pointer_only":7}}],"papers_shown":2,"tasks":[],"tasks_shown":0,"n_tasks":0,"usage_by_year":[{"year":"2020","papers":1},{"year":"2024","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/reglu"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}