Papers › Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

12 Sep 2021arXiv:2109.05422archive 2025-07-28

Chuanxin Tang, Yucheng Zhao, Guangting Wang, Chong Luo, Wenxuan Xie, Wenjun Zeng

Transformers have sprung up in the field of computer vision. In this work, we explore whether the core self-attention module in Transformer is the key to achieving excellent performance in image recognition. To this end, we build an attention-free network called sMLPNet based on the existing MLP-based vision models. Specifically, we replace the MLP module in the token-mixing step with a novel sparse MLP (sMLP) module. For 2D image tokens, sMLP applies 1D MLP along the axial directions and the parameters are shared among rows or columns. By sparse connection and weight sharing, sMLP module significantly reduces the number of model parameters and computational complexity, avoiding the common over-fitting problem that plagues the performance of MLP-like models. When only trained on the ImageNet-1K dataset, the proposed sMLPNet achieves 81.9% top-1 accuracy with only 24M parameters, which is much better than most CNNs and vision Transformers under the same model size constraint. When scaling up to 66M parameters, sMLPNet achieves 83.4% top-1 accuracy, which is on par with the state-of-the-art Swin Transformer. The success of sMLPNet suggests that the self-attention mechanism is not necessarily a silver bullet in computer vision. The code and models are publicly available at https://github.com/microsoft/SPACH

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

microsoft/SPACH officialmentioned in papermentioned on GitHubpytorch report

liuruiyang98/Jittor-MLP mentioned on GitHubjax report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image Classification

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Image Classification	ImageNet	sMLPNet-B (ImageNet-1k)	Number of params	65.9M	#433 of 1060	Archive leaderboard	report
Image Classification	ImageNet	sMLPNet-B (ImageNet-1k)	Top 1 Accuracy	83.4%	#433 of 1060	Archive leaderboard	report
Image Classification	ImageNet	sMLPNet-S (ImageNet-1k)	Number of params	48.6M	#469 of 1060	Archive leaderboard	report
Image Classification	ImageNet	sMLPNet-S (ImageNet-1k)	Top 1 Accuracy	83.1%	#469 of 1060	Archive leaderboard	report
Image Classification	ImageNet	sMLPNet-T (ImageNet-1k)	Number of params	24.1M	#599 of 1060	Archive leaderboard	report
Image Classification	ImageNet	sMLPNet-T (ImageNet-1k)	Top 1 Accuracy	81.9%	#599 of 1060	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention Average Pooling BPE Dense Connections Dropout Global Average Pooling Label Smoothing Layer Normalization Linear Layer MLP-Mixer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Stochastic Depth Swin Transformer Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections