Methods › General › Activation Functions › Squared ReLU
Squared ReLU
Introduced by David R. So et al. in Primer: Searching for Efficient Transformers for Language Modeling
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Squared ReLU is an activation function used in the Primer architecture in the feedforward block of the Transformer layer. It is simply squared ReLU activations.
The effectiveness of higher order polynomials can also be observed in other effective Transformer nonlinearities, such as GLU variants like ReGLU and point-wise activations like approximate GELU. However, squared ReLU has drastically different asymptotics as x →inf compared to the most commonly used activation functions: ReLU, GELU and Swish. Squared ReLU does have significant overlap with ReGLU and in fact is equivalent when ReGLU’s U and V weight matrices are the same and squared ReLU is immediately preceded by a linear transformation with weight matrix U. This leads the authors to believe that squared ReLUs capture the benefits of these GLU variants, while being simpler, without additional parameters, and delivering better quality.
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: P-ReLU; ReLU code space; log-sum-ReLU (LSR); Transformer-Squared; Brownian ReLU (Br-ReLU); Fast Length-Squared Sampling; Squared Neural Family (SNEFY); Height-Augmented ReLU Networks.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 15 · with a code link: 5 · where Syntology ran a sample: 2 (2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (2 of 15 tagged: 2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 15 of 15, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment 20 May 2025 · 0 repositories · arXiv:2505.14667
-
A review of DNA restriction-free overlapping sequence cloning techniques for synthetic biology 6 May 2025 · 0 repositories · arXiv:2505.03681
-
Primer C-VAE: An interpretable deep learning primer design method to detect emerging virus variants 3 Mar 2025 · 0 repositories · arXiv:2503.01459
-
Deriving Activation Functions Using Integration 20 Nov 2024 · 1 repository · arXiv:2411.13010
-
Characteristic Performance Study on Solving Oscillator ODEs via Soft-constrained Physics-informed Neural Network with Small Data 19 Aug 2024 · 1 repository · arXiv:2408.11077
-
The curious case of A31P, a topology-switching mutant of the Repressor of Primer protein : A molecular dynamics study of its folding and misfolding 1 Apr 2024 · 0 repositories · arXiv:2404.01405
-
Brainformers: Trading Simplicity for Efficiency 29 May 2023 · 0 repositories · arXiv:2306.00008
-
The Effects of Political Martyrdom on Election Results: The Assassination of Abe 29 May 2023 · 1 repository · arXiv:2305.18004
-
Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks 23 Jan 2023 · 0 repositories · arXiv:2301.09245
-
N-Grammer: Augmenting Transformers with latent n-grams 13 Jul 2022 · 2 repositories · arXiv:2207.06366Syntology official (archive's flag): 5 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)
-
Piecewise Linear Neural Networks and Deep Learning 18 Jun 2022 · 0 repositories · arXiv:2206.09149
-
Enriching and Characterizing T-Cell Repertoires from 3' Barcoded Single-Cell Whole Transcriptome Amplification Products 21 Mar 2022 · 0 repositories · arXiv:2203.11266
-
Searching for Efficient Transformers for Language Modeling 1 Dec 2021 · 0 repositories
-
N-grammer: Augmenting Transformers with latent n-grams 16 Nov 2021 · 0 repositories
-
Primer: Searching for Efficient Transformers for Language Modeling 17 Sep 2021 · 4 repositories · arXiv:2109.08668Syntology official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
Tasks archive 2025-07-28
18 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections