Papers › Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective

Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective

5 Nov 2024arXiv:2411.03033archive 2025-07-28

Qishuai Wen, Chun-Guang Li

State-of-the-art methods for Transformer-based semantic segmentation typically adopt Transformer decoders that are used to extract additional embeddings from image embeddings via cross-attention, refine either or both types of embeddings via self-attention, and project image embeddings onto the additional embeddings via dot-product. Despite their remarkable success, these empirical designs still lack theoretical justifications or interpretations, thus hindering potentially principled improvements. In this paper, we argue that there are fundamental connections between semantic segmentation and compression, especially between the Transformer decoders and Principal Component Analysis (PCA). From such a perspective, we derive a white-box, fully attentional DEcoder for PrIncipled semantiC segemenTation (DEPICT), with the interpretations as follows: 1) the self-attention operator refines image embeddings to construct an ideal principal subspace that aligns with the supervision and retains most information; 2) the cross-attention operator seeks to find a low-rank approximation of the refined image embeddings, which is expected to be a set of orthonormal bases of the principal subspace and corresponds to the predefined classes; 3) the dot-product operation yields compact representation for image embeddings as segmentation masks. Experiments conducted on dataset ADE20K find that DEPICT consistently outperforms its black-box counterpart, Segmenter, and it is light weight and more robust.

Paper PDF Code

Code

qishuaiwen/depict officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Decoder Segmentation Semantic Segmentation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Semantic Segmentation	ADE20K val	DEPICT-SA (ViT-L 640x640 multi-scale)	mIoU	54.3	#34 of 95	Archive leaderboard	report
Semantic Segmentation	ADE20K val	DEPICT-SA (ViT-L 640x640 single-scale)	mIoU	52.9	#42 of 95	Archive leaderboard	report
Semantic Segmentation	Cityscapes val	DEPICT-SA (ViT-L multi-scale)	mIoU	81.0	#47 of 99	Archive leaderboard	report
Semantic Segmentation	Cityscapes val	DEPICT-SA (ViT-L single-scale)	mIoU	78.8	#59 of 99	Archive leaderboard	report
Semantic Segmentation	PASCAL Context	DEPICT-SA (ViT-L multi-scale)	mIoU	58.6	#18 of 66	Archive leaderboard	report
Semantic Segmentation	PASCAL Context	DEPICT-SA (ViT-L single-scale)	mIoU	57.9	#20 of 66	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

ADOPT Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection SET Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections