Papers › Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

12 Sep 2021arXiv:2109.05422archive 2025-07-28

Chuanxin Tang, Yucheng Zhao, Guangting Wang, Chong Luo, Wenxuan Xie, Wenjun Zeng

Transformers have sprung up in the field of computer vision. In this work, we explore whether the core self-attention module in Transformer is the key to achieving excellent performance in image recognition. To this end, we build an attention-free network called sMLPNet based on the existing MLP-based vision models. Specifically, we replace the MLP module in the token-mixing step with a novel sparse MLP (sMLP) module. For 2D image tokens, sMLP applies 1D MLP along the axial directions and the parameters are shared among rows or columns. By sparse connection and weight sharing, sMLP module significantly reduces the number of model parameters and computational complexity, avoiding the common over-fitting problem that plagues the performance of MLP-like models. When only trained on the ImageNet-1K dataset, the proposed sMLPNet achieves 81.9% top-1 accuracy with only 24M parameters, which is much better than most CNNs and vision Transformers under the same model size constraint. When scaling up to 66M parameters, sMLPNet achieves 83.4% top-1 accuracy, which is on par with the state-of-the-art Swin Transformer. The success of sMLPNet suggests that the self-attention mechanism is not necessarily a silver bullet in computer vision. The code and models are publicly available at https://github.com/microsoft/SPACH

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

microsoft/SPACH officialmentioned in papermentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to... (arXiv:2201.10801; that paper's own run record: 6 ran (of which 6 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 6 samples that ran constructed an object rather than computing a result); nothing checks that they implement this paper's method. report
liuruiyang98/Jittor-MLP mentioned on GitHubjaxSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to S²-MLPv2: Improved Spatial-Shift MLP Architecture for Vision (arXiv:2108.01072; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image Classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Image Classification ImageNet sMLPNet-B (ImageNet-1k) Number of params 65.9M #433 of 1060 Archive leaderboard report
Image Classification ImageNet sMLPNet-B (ImageNet-1k) Top 1 Accuracy 83.4% #433 of 1060 Archive leaderboard report
Image Classification ImageNet sMLPNet-S (ImageNet-1k) Number of params 48.6M #469 of 1060 Archive leaderboard report
Image Classification ImageNet sMLPNet-S (ImageNet-1k) Top 1 Accuracy 83.1% #469 of 1060 Archive leaderboard report
Image Classification ImageNet sMLPNet-T (ImageNet-1k) Number of params 24.1M #599 of 1060 Archive leaderboard report
Image Classification ImageNet sMLPNet-T (ImageNet-1k) Top 1 Accuracy 81.9% #599 of 1060 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionAverage PoolingBPEDense ConnectionsDropoutGlobal Average PoolingLabel SmoothingLayer NormalizationLinear LayerMLP-MixerMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSoftmaxStochastic DepthSwin TransformerTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections