Methods › Natural Language Processing › Transformers › Sparse Transformer
Sparse Transformer
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
A Sparse Transformer is a Transformer based architecture which utilises sparse factorizations of the attention matrix to reduce time/memory to O(n √(n)). Other changes to the Transformer architecture include: (a) a restructured residual block and weight initialization, (b) A set of sparse attention kernels which efficiently compute subsets of the attention matrix, (c) recomputation of attention weights during the backwards pass to reduce memory usage
In Syntology the paper matched by title, the paper Syntology's graph keys as openreview_Hye87grYDH, held, no code (no_code; paper_only). The lookup matched a paper by its title, the paper Syntology's graph keys as openreview_Hye87grYDH, not a method entry. The archive credits no paper with an arXiv id for this method, so whether this answer (about the paper Syntology's graph keys as openreview_Hye87grYDH) is about this method is not checked; it does not say whether this method has code.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 47 · with a code link: 32 · where Syntology ran a sample: 13 (8 with a run with no instrument failure, 5 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (13 of 47 tagged: 8 with a run with no instrument failure, 5 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 47 of 47, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Pyramid Sparse Transformer: Enhancing Multi-Scale Feature Fusion with Dynamic Token Selection 19 May 2025 · 0 repositories · arXiv:2505.12772
-
Tractable Transformers for Flexible Conditional Generation 11 Feb 2025 · 0 repositories · arXiv:2502.07616
-
DeepGate4: Efficient and Effective Representation Learning for Circuit Design at Scale 2 Feb 2025 · 1 repository · arXiv:2502.01681Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
SPARTAN: A Sparse Transformer Learning Local Causation 11 Nov 2024 · 0 repositories · arXiv:2411.06890
-
Extra Global Attention Designation Using Keyword Detection in Sparse Transformer Architectures 11 Oct 2024 · 0 repositories · arXiv:2410.08971
-
Efficient and Scalable Point Cloud Generation with Sparse Point-Voxel Diffusion Models 12 Aug 2024 · 2 repositories · arXiv:2408.06145
-
Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget 22 Jul 2024 · 1 repository · arXiv:2407.15811Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples)
-
HDT: Hierarchical Document Transformer 11 Jul 2024 · 0 repositories · arXiv:2407.08330
-
Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Heads 27 Jun 2024 · 1 repository · arXiv:2406.19391
-
A Mixture of Experts Approach to 3D Human Motion Prediction 9 May 2024 · 1 repository · arXiv:2405.06088
-
Scene Adaptive Sparse Transformer for Event-based Object Detection 2 Apr 2024 · 1 repository · arXiv:2404.01882Syntology official (archive's flag): 19 ran · 19 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 13 where Syntology's instrument failed) · 4 unverified (of 23 harvested samples)
-
SMTF: Sparse transformer with multiscale contextual fusion for medical image segmentation 24 Mar 2024 · 1 repository
-
Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration 9 Mar 2024 · 0 repositories · arXiv:2403.05906
-
Do Efficient Transformers Really Save Computation? 21 Feb 2024 · 0 repositories · arXiv:2402.13934
-
Adapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Image Restoration 1 Jan 2024 · 1 repository
-
Towards More Unified In-context Visual Understanding 5 Dec 2023 · 0 repositories · arXiv:2312.02520
-
SPION: Layer-Wise Sparse Training of Transformer via Convolutional Flood Filling 22 Sep 2023 · 0 repositories · arXiv:2309.12578
-
SparseSwin: Swin Transformer with Sparse Transformer Block 11 Sep 2023 · 1 repository · arXiv:2309.05224
-
From Sparse to Soft Mixtures of Experts 2 Aug 2023 · 5 repositories · arXiv:2308.00951Syntology official (archive's flag): 3 ran · 29 ran (of which 5 constructed an object rather than computing a result; 18 with no instrument failure: 3 honoured, 4 violated, 11 with no contract checked; 11 where Syntology's instrument failed) · 4 unverified (of 33 harvested samples) · 8 pointer-only (licence)
-
LongCoder: A Long-Range Pre-trained Language Model for Code Completion 26 Jun 2023 · 1 repository · arXiv:2306.14893Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Learning A Sparse Transformer Network for Effective Image Deraining 21 Mar 2023 · 1 repository · arXiv:2303.11950Syntology official (archive's flag): 8 ran · 8 ran (of which 8 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified; every one of the 8 samples that ran constructed an object rather than computing a result (of 11 harvested samples) · 11 pointer-only (licence)
-
Sampled Transformer for Point Sets 28 Feb 2023 · 0 repositories · arXiv:2302.14346
-
Generating a Structured Summary of Numerous Academic Papers: Dataset and Method 9 Feb 2023 · 1 repository · arXiv:2302.04580
-
Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences 21 Oct 2022 · 1 repository · arXiv:2210.11794
-
Efficient Quantized Sparse Matrix Operations on Tensor Cores 14 Sep 2022 · 1 repository · arXiv:2209.06979
-
DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation 13 Jul 2022 · 1 repository · arXiv:2207.06124Syntology official (archive's flag): 5 ran · 5 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 8 harvested samples)
-
SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning 11 Jul 2022 · 2 repositories · arXiv:2207.04606Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 2 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)
-
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale 30 Jun 2022 · 2 repositories · arXiv:2207.00032
-
Video Sparse Transformer With Attention-Guided Memory for Video Object Detection 17 Jun 2022 · 1 repository
-
What Dense Graph Do You Need for Self-Attention? 27 May 2022 · 1 repository · arXiv:2205.14014
-
ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention 23 Mar 2022 · 0 repositories · arXiv:2203.12276
-
Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction 9 Mar 2022 · 1 repository · arXiv:2203.04845Syntology official (archive's flag): 13 ran · 13 ran (of which 8 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 16 harvested samples)
-
Flow-Guided Sparse Transformer for Video Deblurring 6 Jan 2022 · 1 repository · arXiv:2201.01893Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Embracing Single Stride 3D Object Detector with Sparse Transformer 13 Dec 2021 · 2 repositories · arXiv:2112.06375
-
ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention 16 Nov 2021 · 0 repositories
-
Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting 13 Oct 2021 · 1 repository · arXiv:2110.08255
-
ERNIE-SPARSE: Robust Efficient Transformer Through Hierarchically Unifying Isolated Information 29 Sep 2021 · 0 repositories
-
Cluster-Former: Clustering-based Sparse Transformer for Question Answering 1 Jan 2021 · 0 repositories
-
VisualSparta: An Embarrassingly Simple Approach to Large-scale Text-to-Image Search with Weighted Bag-of-words 1 Jan 2021 · 1 repository · arXiv:2101.00265
-
Adversarial Sparse Transformer for Time Series Forecasting 1 Dec 2020 · 1 repository
-
Scene Graph Modification Based on Natural Language Commands 6 Oct 2020 · 1 repository · arXiv:2010.02591
-
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding 13 Sep 2020 · 0 repositories · arXiv:2009.06097
-
Sparsifying Transformer Models with Trainable Representation Pooling 10 Sep 2020 · 1 repository · arXiv:2009.05169
-
GMAT: Global Memory Augmentation for Transformers 5 Jun 2020 · 1 repository · arXiv:2006.03274
-
Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection 25 Dec 2019 · 2 repositories · arXiv:1912.11637Syntology official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 1 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Adaptively Sparse Transformers 30 Aug 2019 · 3 repositories · arXiv:1909.00015Syntology official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Generating Long Sequences with Sparse Transformers 23 Apr 2019 · 7 repositories · arXiv:1904.10509Syntology community repositories only · 5 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)
Tasks archive 2025-07-28
20 shown of 80 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections