Methods › General › Feedforward Networks › Spatial Gating Unit
Spatial Gating Unit
Introduced by Hanxiao Liu et al. in Pay Attention to MLPs
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Spatial Gating Unit, or SGU, is a gating unit used in the gMLP architecture to captures spatial interactions. To enable cross-token interactions, it is necessary for the layer s(·) to contain a contraction operation over the spatial dimension. The layer s(·) is formulated as the output of linear gating:
s(Z)=Z ⊙f_(W, b)(Z)
where ⊙ denotes element-wise multiplication. For training stability, the authors find it critical to initialize W as near-zero values and b as ones, meaning that f_(W, b)(Z) ≈1 and therefore s(Z) ≈Z at the beginning of training. This initialization ensures each gMLP block behaves like a regular FFN at the early stage of training, where each token is processed independently, and only gradually injects spatial information across tokens during the course of learning.
The authors find it further effective to split Z into two independent parts (Z₁, Z₂) along the channel dimension for the gating function and for the multiplicative bypass:
s(Z)=Z₁ ⊙f_(W, b)(Z₂)
They also normalize the input to f_(W, b) which empirically improved the stability of large NLP models.
Papers archive 2025-07-28
9 shown of 9, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Image Super-resolution Reconstruction Network based on Enhanced Swin Transformer via Alternating Aggregation of Local-Global Features 30 Dec 2023 · 0 repositories · arXiv:2401.00241
-
SigVIC: Spatial Importance Guided Variable-Rate Image Compression 16 Mar 2023 · 0 repositories · arXiv:2303.09112
-
gSwin: Gated MLP Vision Model with Hierarchical Structure of Shifted Window 24 Aug 2022 · 0 repositories · arXiv:2208.11718
-
A Proposal of Multi-Layer Perceptron with Graph Gating Unit for Graph Representation Learning and its Application to Surrogate Model for FEM 11 Jul 2022 · 1 repository
-
Are We Really Making Much Progress in Text Classification? A Comparative Review 8 Apr 2022 · 1 repository · arXiv:2204.03954
-
Efficient Language Modeling with Sparse all-MLP 14 Mar 2022 · 0 repositories · arXiv:2203.06850
-
Convolutional Gated MLP: Combining Convolutions & gMLP 6 Nov 2021 · 0 repositories · arXiv:2111.03940
-
CycleMLP: A MLP-like Architecture for Dense Prediction 21 Jul 2021 · 8 repositories · arXiv:2107.10224Syntology ran 8 of 15 samples · 7 unverified · 1 pointer-only (licence)
-
Pay Attention to MLPs 17 May 2021 · 20 repositories · arXiv:2105.08050Syntology ran 34 of 44 samples · 10 unverified · 11 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 35 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections