Papers › Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning

Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning

9 Dec 2022CVPR 2023 1arXiv:2212.04994archive 2025-07-28

Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip H. S. Torr, Ser-Nam Lim

We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabulary zero-shot segmentation on 4 different segmentation benchmarks: Pascal VOC, Pascal Context, COCO Stuff and ADE20K. Furthermore, we show that PACL is also applicable to image-level predictions and when used with a CLIP backbone, provides a general improvement in zero-shot classification accuracy compared to CLIP, across a suite of 12 image classification datasets.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

paulcouairon/diffcut mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Contrastive Learning Image Classification Open Vocabulary Semantic Segmentation Open-Vocabulary Semantic Segmentation Segmentation Semantic Segmentation Zero Shot Segmentation Zero-Shot Learning image-classification

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Open Vocabulary Semantic Segmentation	ADE20K-150	PACL	mIoU	31.4	#15 of 23	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	Cityscape-171	PACL	mIoU	38.8	#1 of 1	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	PASCAL Context-59	PACL	mIoU	50.1	#16 of 24	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	PascalVOC-20	PACL	mIoU	72.3	#18 of 20	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

CLIP Contrastive Learning

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections