Papers › SparseSwin: Swin Transformer with Sparse Transformer Block

SparseSwin: Swin Transformer with Sparse Transformer Block

11 Sep 2023arXiv:2309.05224archive 2025-07-28

Krisna Pinasthika, Blessius Sheldo Putra Laksono, Riyandi Banovbi Putera Irsal, Syifa Hukma Shabiyya, Novanto Yudistira

Advancements in computer vision research have put transformer architecture as the state of the art in computer vision tasks. One of the known drawbacks of the transformer architecture is the high number of parameters, this can lead to a more complex and inefficient algorithm. This paper aims to reduce the number of parameters and in turn, made the transformer more efficient. We present Sparse Transformer (SparTa) Block, a modified transformer block with an addition of a sparse token converter that reduces the number of tokens used. We use the SparTa Block inside the Swin T architecture (SparseSwin) to leverage Swin capability to downsample its input and reduce the number of initial tokens to be calculated. The proposed SparseSwin model outperforms other state of the art models in image classification with an accuracy of 86.96%, 97.43%, and 85.35% on the ImageNet100, CIFAR10, and CIFAR100 datasets respectively. Despite its fewer parameters, the result highlights the potential of a transformer architecture using a sparse token converter with a limited number of tokens to optimize the use of the transformer and improve its performance.

Paper PDF Code

Code

krisnapinasthika/sparseswin officialmentioned in paperpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image Classification image-classification

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Image Classification	CIFAR-10	SparseSwin	Percentage correct	97.43	#83 of 265	Archive leaderboard	report
Image Classification	CIFAR-100	SparseSwin	PARAMS	17.58M	#64 of 211	Archive leaderboard	report
Image Classification	CIFAR-100	SparseSwin	Percentage correct	85.35	#64 of 211	Archive leaderboard	report
Image Classification	ImageNet-100 (TEMI Split)	SparseSwin with L2	Params	17.58M	#1 of 2	Archive leaderboard	report
Image Classification	ImageNet-100 (TEMI Split)	SparseSwin with L2	Percentage correct	86.96	#1 of 2	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention Attention Dropout BPE Cosine Annealing Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Linear Warmup With Cosine Annealing Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Sparse Transformer Transformer Weight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections