Efficient ViTs

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

27 shown of 27 papers with code (32 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Training data-efficient image transformers & distillation through attention
  23 Dec 2020 40 repositories listed Syntology ran 12 of 19 samples · 7 unverified · 3 pointer-only (licence)
  In this work, we produce a competitive convolution-free transformer by training on Imagenet only.
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
  22 Apr 2021 7 repositories listed Syntology ran 3 of 5 samples · 2 unverified · 3 pointer-only (licence)
  In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs).
• Token Merging: Your ViT But Faster
  17 Oct 2022 5 repositories listed Syntology ran 4 of 5 samples · 1 unverified · 4 pointer-only (licence)
  Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.
• Fast Vision Transformers with HiLo Attention
  26 May 2022 5 repositories listed Syntology ran 2 of 13 samples · 11 unverified
  Therefore, we propose to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, where one group encodes high frequencies via self-attention within each local window,…
• Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inference
  18 Nov 2022 4 repositories listed
  Vision Transformers (ViTs) have shown impressive performance but still require a high computation cost as compared to convolutional neural networks (CNNs), one reason is that ViTs' attention measures global similarities…
• Pruning Self-attentions into Convolutional Layers in Single Path
  23 Nov 2021 3 repositories listed Syntology ran 0 of 7 samples · 7 unverified
  Relying on the single-path space, we introduce learnable binary gates to encode the operation choices in MSA layers.
• PPT: Token Pruning and Pooling for Efficient Vision Transformers
  3 Oct 2023 2 repositories listed Syntology ran 5 of 7 samples · 2 unverified
  Vision Transformers (ViTs) have emerged as powerful models in the field of computer vision, delivering superior performance across various vision tasks.
• MDViT: Multi-domain Vision Transformer for Small Medical Image Segmentation Datasets
  5 Jul 2023 2 repositories listed
  Naivly combining datasets from different domains can result in negative knowledge transfer (NKT), i.
• Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations
  16 Feb 2022 2 repositories listed Syntology ran 6 of 15 samples · 9 unverified · 12 pointer-only (licence)
  Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images.
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
  3 Jun 2021 2 repositories listed Syntology ran 6 of 9 samples · 3 unverified · 1 pointer-only (licence)
  Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input.
• Scalable Vision Transformers with Hierarchical Pooling
  19 Mar 2021 2 repositories listed Syntology ran 1 of 2 samples · 1 unverified
  However, the routine of the current ViT model is to maintain a full-length patch sequence during inference, which is redundant and lacks hierarchical representation.
• Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer
  6 May 2024 1 repository listed
  Motivated by the huge success of Transformers in the field of natural language processing (NLP), Vision Transformers (ViTs) have been rapidly developed and achieved remarkable performance in various computer vision…
• Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers
  15 Mar 2024 1 repository listed Syntology ran 3 of 5 samples · 2 unverified
  Here, we argue that token fusion needs to consider diverse relations between tokens to minimize information loss.
• GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation
  6 Nov 2023 1 repository listed Syntology ran 1 of 2 samples · 1 unverified
  Vision Transformers (ViTs) have revolutionized the field of computer vision, yet their deployments on resource-constrained devices remain challenging due to high computational demands.
• Learned Thresholds Token Merging and Pruning for Vision Transformers
  20 Jul 2023 1 repository listed Syntology ran 4 of 5 samples · 1 unverified · 5 pointer-only (licence)
  Vision transformers have demonstrated remarkable success in a wide range of computer vision tasks over the last years.
• ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer
  10 Jun 2023 1 repository listed Syntology ran 8 of 20 samples · 12 unverified
  To marry the best of both worlds, we further propose a new mixture of experts (MoE) framework to reparameterize MLPs by taking multiplication or its primitives as experts, e.
• DiffRate : Differentiable Compression Rate for Efficient Vision Transformers
  29 May 2023 1 repository listed
  Token compression aims to speed up large-scale vision transformers (e.
• Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
  21 Apr 2023 1 repository listed Syntology ran 2 of 8 samples · 6 unverified
  Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy.
• Making Vision Transformers Efficient from A Token Sparsification View
  15 Mar 2023 1 repository listed Syntology ran 2 of 4 samples · 2 unverified
  In this work, we propose a novel Semantic Token ViT (STViT), for efficient global and local vision transformers, which can also be revised to serve as backbone for downstream tasks.
• Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers
  21 Nov 2022 1 repository listed
  In this paper, we emphasize the cruciality of diverse global semantics and propose an efficient token decoupling and merging method that can jointly consider the token importance and diversity for token pruning.
• Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
  28 Sep 2022 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  The learnable thresholds are optimized in budget-aware training to balance accuracy and complexity, performing the corresponding pruning configurations for different input instances.
• SPViT: Enabling Faster Vision Transformers via Soft Token Pruning
  27 Dec 2021 1 repository listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  Moreover, our framework can guarantee the identified model to meet resource specifications of mobile devices and FPGA, and even achieve the real-time execution of DeiT-T on mobile platforms.
• AdaViT: Adaptive Tokens for Efficient Vision Transformer
  14 Dec 2021 1 repository listed Syntology ran 0 of 7 samples · 7 unverified · 7 pointer-only (licence)
  A-ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
• Adaptive Token Sampling For Efficient Vision Transformers
  30 Nov 2021 1 repository listed Syntology ran 1 of 5 samples · 4 unverified · 5 pointer-only (licence)
  Since ATS is a parameter-free module, it can be added to the off-the-shelf pre-trained vision transformers as a plug and play module, thus reducing their GFLOPs without any additional training.
• Global Vision Transformer Pruning with Hessian-Aware Saliency
  10 Oct 2021 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  This work aims on challenging the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage, where we redistribute the parameters both across…
• Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
  3 Aug 2021 1 repository listed Syntology ran 5 of 9 samples · 4 unverified
  Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue.
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
  8 Jun 2021 1 repository listed Syntology ran 2 of 7 samples · 5 unverified
  For example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data, architecture), improves 0.

Syntology lines on 22 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.