Papers › kMaX-DeepLab: k-means Mask Transformer

kMaX-DeepLab: k-means Mask Transformer

8 Jul 2022arXiv:2207.04044archive 2025-07-28

Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, Liang-Chieh Chen

The rise of transformers in vision tasks not only advances network backbone designs, but also starts a brand-new page to achieve end-to-end image recognition (e.g., object detection and panoptic segmentation). Originated from Natural Language Processing (NLP), transformer architectures, consisting of self-attention and cross-attention, effectively learn long-range interactions between elements in a sequence. However, we observe that most existing transformer-based vision models simply borrow the idea from NLP, neglecting the crucial difference between languages and images, particularly the extremely large sequence length of spatially flattened pixel features. This subsequently impedes the learning in cross-attention between pixel features and object queries. In this paper, we rethink the relationship between pixels and object queries and propose to reformulate the cross-attention learning as a clustering process. Inspired by the traditional k-means clustering algorithm, we develop a k-means Mask Xformer (kMaX-DeepLab) for segmentation tasks, which not only improves the state-of-the-art, but also enjoys a simple and elegant design. As a result, our kMaX-DeepLab achieves a new state-of-the-art performance on COCO val set with 58.0% PQ, Cityscapes val set with 68.4% PQ, 44.0% AP, and 83.5% mIoU, and ADE20K val set with 50.9% PQ and 55.2% mIoU without test-time augmentation or external dataset. We hope our work can shed some light on designing transformers tailored for vision tasks. TensorFlow code and models are available at https://github.com/google-research/deeplab2 A PyTorch re-implementation is also available at https://github.com/bytedance/kmax-deeplab

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

bytedance/kmax-deeplab officialmentioned in papermentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (arXiv:2206.08948; that paper's own run record: 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
google-research/deeplab2 officialmentioned in papermentioned on GitHubtfApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to PolyMaX: General Dense Prediction with Mask Transformer (arXiv:2311.05770; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
cy-xu/spatially_aware_ai mentioned on GitHubpytorchAGPL-3.0Syntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

ClusteringObject DetectionPanoptic SegmentationSemantic Segmentation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Panoptic Segmentation ADE20K val kMaX-DeepLab (ConvNeXt-L, single-scale, 1281x1281) AP - #10 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ConvNeXt-L, single-scale, 1281x1281) PQ 50.9 #10 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ConvNeXt-L, single-scale, 1281x1281) mIoU 55.2 #10 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ConvNeXt-L, single-scale, 641x641) AP - #17 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ConvNeXt-L, single-scale, 641x641) PQ 48.7 #17 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ConvNeXt-L, single-scale, 641x641) mIoU 54.8 #17 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ResNet50, single-scale, 1281x1281) AP - #20 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ResNet50, single-scale, 1281x1281) PQ 42.3 #20 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ResNet50, single-scale, 1281x1281) mIoU 45.3 #20 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ResNet50, single-scale, 641x641) AP - #21 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ResNet50, single-scale, 641x641) PQ 41.5 #21 of 25 Archive leaderboard report
Panoptic Segmentation ADE20K val kMaX-DeepLab (ResNet50, single-scale, 641x641) mIoU 45.0 #21 of 25 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale, pseudo-labels) PQ 58.1 #11 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale, pseudo-labels) PQst 48.8 #11 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale, pseudo-labels) PQth 64.3 #11 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale, drop query with 256 queries) PQ 58.0 #13 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale, drop query with 256 queries) PQst 48.6 #13 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale, drop query with 256 queries) PQth 64.2 #13 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale) PQ 57.9 #15 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale) PQst 48.6 #15 of 31 Archive leaderboard report
Panoptic Segmentation COCO minival kMaX-DeepLab (single-scale) PQth 64.0 #15 of 31 Archive leaderboard report
Panoptic Segmentation COCO test-dev kMaX-DeepLab (single-scale) PQ 58.5 #2 of 38 Archive leaderboard report
Panoptic Segmentation COCO test-dev kMaX-DeepLab (single-scale) PQst 49.0 #2 of 38 Archive leaderboard report
Panoptic Segmentation COCO test-dev kMaX-DeepLab (single-scale) PQth 64.8 #2 of 38 Archive leaderboard report
Panoptic Segmentation Cityscapes test kMaX-DeepLab (single-scale) PQ 66.2 #5 of 10 Archive leaderboard report
Panoptic Segmentation Cityscapes val kMaX-DeepLab (single-scale) AP 44.0 #8 of 37 Archive leaderboard report
Panoptic Segmentation Cityscapes val kMaX-DeepLab (single-scale) PQ 68.4 #8 of 37 Archive leaderboard report
Panoptic Segmentation Cityscapes val kMaX-DeepLab (single-scale) mIoU 83.5 #8 of 37 Archive leaderboard report
Semantic Segmentation Cityscapes test kMaX-DeepLab (ConvNeXt-L, fine only) Mean IoU (class) 83.2% #18 of 105 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

k-Means Clustering

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections