Papers › Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

31 Dec 2020CVPR 2021 1arXiv:2012.15840archive 2025-07-28

Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, Li Zhang

Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (ie, without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

fudan-zvg/SETR officialmentioned on GitHubpytorchSyntology: not harvested report
920232796/setr-pytorch mentioned on GitHubpytorchSyntology: not harvested report
gupta-abhay/setr-pytorch mentioned on GitHubpytorchSyntology: not harvested report
BR-IDL/PaddleViT paddleSyntology: not harvested report
PaddlePaddle/PaddleSeg paddleSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow (arXiv:2109.09406; that paper's own run record: 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

DecoderMedical Image SegmentationSegmentationSemantic Segmentation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Medical Image Segmentation Synapse multi-organ CT SETR Avg DSC 79.60 #20 of 23 Archive leaderboard report
Semantic Segmentation ADE20K SETR-MLA (160k, MS) Validation mIoU 50.28 #116 of 235 Archive leaderboard report
Semantic Segmentation Cityscapes test SETR-PUP++ Mean IoU (class) 81.64% #37 of 105 Archive leaderboard report
Semantic Segmentation Cityscapes val SETR-PUP (80k, MS) mIoU 82.15 #37 of 99 Archive leaderboard report
Semantic Segmentation DADA-seg SETR (PUP, Transformer-Large) mIoU 31.8 #4 of 28 Archive leaderboard report
Semantic Segmentation DADA-seg SETR (MLA, Transformer-Large) mIoU 30.4 #5 of 28 Archive leaderboard report
Semantic Segmentation DensePASS SETR (PUP, Transformer-L) mIoU 35.7% #18 of 36 Archive leaderboard report
Semantic Segmentation DensePASS SETR (MLA, Transformer-L) mIoU 35.6% #19 of 36 Archive leaderboard report
Semantic Segmentation FoodSeg103 SeTR-MLA (ViT-16/B) mIoU 45.1 #2 of 7 Archive leaderboard report
Semantic Segmentation FoodSeg103 SeTR-Naive (ViT-16/B) mIoU 41.3 #5 of 7 Archive leaderboard report
Semantic Segmentation PASCAL Context SETR-MLA (16, 80k, MS) mIoU 55.83 #27 of 66 Archive leaderboard report
Semantic Segmentation UrbanLF SETR (ViT-Large) mIoU (Real) 77.74 #9 of 14 Archive leaderboard report
Semantic Segmentation UrbanLF SETR (ViT-Large) mIoU (Syn) 77.69 #9 of 14 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AttentionConvolutionDense ConnectionsFCNLayer NormalizationLinear LayerMax PoolingMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSETRSoftmax

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections