Papers › Generalized Decoding for Pixel, Image, and Language
Generalized Decoding for Pixel, Image, and Language
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, JianFeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, Jianfeng Gao
We present X-Decoder, a generalized decoding model that can predict pixel-level segmentation and language tokens seamlessly. X-Decodert takes as input two types of queries: (i) generic non-semantic queries and (ii) semantic queries induced from text inputs, to decode different pixel-level and token-level outputs in the same semantic space. With such a novel design, X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks. Further, our design enables seamless interactions across tasks at different granularities and brings mutual benefits by learning a common and rich pixel-level visual-semantic understanding space, without any pseudo-labeling. After pretraining on a mixed set of a limited amount of segmentation data and millions of image-text pairs, X-Decoder exhibits strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings. Notably, it achieves (1) state-of-the-art results on open-vocabulary segmentation and referring segmentation on eight datasets; (2) better or competitive finetuned performance to other generalist and specialist models on segmentation and VL tasks; and (3) flexibility for efficient finetuning and novel task composition (e.g., referring captioning and image editing). Code, demo, video, and visualization are available at https://x-decoder-vl.github.io.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Instance Segmentation | ADE20K val | X-Decoder (Davit-d5, Deform, single-scale, 1280x1280) | AP | 38.7 | #5 of 14 | Archive leaderboard | report |
| Instance Segmentation | ADE20K val | X-Decoder (Davit-d5, Deform, single-scale, 1280x1280) | APL | 59.6 | #5 of 14 | Archive leaderboard | report |
| Instance Segmentation | ADE20K val | X-Decoder (Davit-d5, Deform, single-scale, 1280x1280) | APM | 43.3 | #5 of 14 | Archive leaderboard | report |
| Instance Segmentation | ADE20K val | X-Decoder (Davit-d5, Deform, single-scale, 1280x1280) | APS | 18.9 | #5 of 14 | Archive leaderboard | report |
| Instance Segmentation | ADE20K val | X-Decoder (L) | AP | 35.8 | #9 of 14 | Archive leaderboard | report |
| Panoptic Segmentation | ADE20K val | X-Decoder (Davit-d5, Deform, single-scale, 1280x1280) | AP | 38.7 | #6 of 25 | Archive leaderboard | report |
| Panoptic Segmentation | ADE20K val | X-Decoder (Davit-d5, Deform, single-scale, 1280x1280) | PQ | 52.4 | #6 of 25 | Archive leaderboard | report |
| Panoptic Segmentation | ADE20K val | X-Decoder (Davit-d5, Deform, single-scale, 1280x1280) | mIoU | 59.1 | #6 of 25 | Archive leaderboard | report |
| Panoptic Segmentation | ADE20K val | X-Decoder (L) | AP | 35.8 | #15 of 25 | Archive leaderboard | report |
| Panoptic Segmentation | ADE20K val | X-Decoder (L) | PQ | 49.6 | #15 of 25 | Archive leaderboard | report |
| Panoptic Segmentation | ADE20K val | X-Decoder (L) | mIoU | 58.1 | #15 of 25 | Archive leaderboard | report |
| Referring Expression Segmentation | RefCOCOg-val | X-Decoder (Davit-d5) | Overall IoU | 64.6 | #17 of 23 | Archive leaderboard | report |
| Zero Shot Segmentation | Segmentation in the Wild | SGinW_Team (X-Decoder-L) | Mean AP | 32.2 | #9 of 12 | Archive leaderboard | report |
| Zero Shot Segmentation | Segmentation in the Wild | SGinW_Team (X-Decoder-B) | Mean AP | 27.7 | #10 of 12 | Archive leaderboard | report |
| Zero Shot Segmentation | Segmentation in the Wild | SGinW_Team (X-Decoder-L-IN21K) | Mean AP | 26.6 | #11 of 12 | Archive leaderboard | report |
| Zero Shot Segmentation | Segmentation in the Wild | SGinW_Team (X-Decoder-T) | Mean AP | 22.6 | #12 of 12 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections