{"url":"/method/geglu","slug":"geglu","name":"GeGLU","full_name":"GeGLU","full_name_withheld":false,"description_markdown":"**GeGLU** is an activation function which is a variant of [GLU](https://paperswithcode.com/method/glu). The definition is as follows:\r\n\r\n$$ \\text{GeGLU}\\left(x, W, V, b, c\\right) = \\text{GELU}\\left(xW + b\\right) \\otimes \\left(xV + c\\right) $$","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2002.05202v1","title":"GLU Variants Improve Transformer","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Activation Functions","url":"/methods/category/activation-functions","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/turbo-sparse-achieving-llm-sota-performance","title":"Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters","date":"2024-06-10","arxiv_id":"2406.05955","n_code_links":1,"syntology":null},{"paper":"/paper/glu-variants-improve-transformer","title":"GLU Variants Improve Transformer","date":"2020-02-12","arxiv_id":"2002.05202","n_code_links":27,"syntology":{"ran":9,"of":9,"unverified":0,"pointer_only":7}}],"papers_shown":2,"tasks":[{"task":"/task/mixture-of-experts","name":"Mixture-of-Experts","papers":1}],"tasks_shown":1,"n_tasks":1,"usage_by_year":[{"year":"2020","papers":1},{"year":"2024","papers":1}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/geglu"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}