Papers › ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

19 Mar 2021arXiv:2103.10697archive 2025-07-28

Stéphane d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, Levent Sagun

Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ``soft" convolutional inductive bias. We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analysing how it is escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

facebookresearch/convit officialmentioned in papermentioned on GitHubpytorchSyntology: not harvested report
SforAiDl/vformer mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Vision Transformers for Dense Prediction (arXiv:2103.13413; that paper's own run record: 65 ran (of which 19 constructed an object rather than computing a result; 32 with no instrument failure: 0 honoured, 0 violated, 32 with no contract checked; 33 where Syntology's instrument failed) · 51 unverified) and ViViT: A Video Vision Transformer (arXiv:2103.15691; that paper's own run record: 14 ran (of which 10 constructed an object rather than computing a result; 13 with no instrument failure: 1 honoured, 0 violated, 12 with no contract checked; 1 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report
facebookresearch/vissl mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles (arXiv:1603.09246; that paper's own run record: 10 ran (of which 6 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified), Unsupervised Representation Learning by Predicting Image Rotations (arXiv:1803.07728; that paper's own run record: 16 ran (of which 7 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 2 where Syntology's instrument failed) · 6 unverified), BEiT: BERT Pre-Training of Image Transformers (arXiv:2106.08254; that paper's own run record: 6 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
lyqcom/convit mentioned on GitHubmindsporeSyntology: not harvested report
rwightman/pytorch-image-models mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (arXiv:2006.08217; that paper's own run record: 4 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
MS-Mind/MS-Code-02 mindsporeSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications (arXiv:2206.10589; that paper's own run record: 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 1 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report
Mind23-2/MindCode-20 mindsporeSyntology: not harvested report
mindspore-courses/External-Attention-MindSpore mindsporeSyntology: not harvested report
mindspore-ecosystem/mindcv mindsporenot reachable when probed 2026-09-17 — repositories for recent papers often appear after camera-readySyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image ClassificationInductive Biasimage-classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Image Classification ImageNet ConViT-B+ GFLOPs 30 #534 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-B+ Number of params 152M #534 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-B+ Top 1 Accuracy 82.5% #534 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-B GFLOPs 17 #541 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-B Number of params 86M #541 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-B Top 1 Accuracy 82.4% #541 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-S+ GFLOPs 10 #565 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-S+ Number of params 48M #565 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-S+ Top 1 Accuracy 82.2% #565 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-S GFLOPs 5.4 #650 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-S Number of params 27M #650 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-S Top 1 Accuracy 81.3% #650 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-Ti+ GFLOPs 2 #905 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-Ti+ Number of params 10M #905 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-Ti+ Top 1 Accuracy 76.7% #905 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-Ti GFLOPs 1 #990 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-Ti Number of params 6M #990 of 1060 Archive leaderboard report
Image Classification ImageNet ConViT-Ti Top 1 Accuracy 73.1% #990 of 1060 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: ConViT, GPSA

AttentionAttention DropoutConViTDeiTDense ConnectionsDropoutFeedforward NetworkGPSALayer NormalizationLinear LayerMulti-Head AttentionResidual ConnectionSoftmax

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections