Papers › LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

4 Dec 2021CVPR 2022 1arXiv:2112.02244archive 2025-07-28

Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, Philip H. S. Torr

Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image. A paradigm for tackling this problem is to leverage a powerful vision-language ("cross-modal") decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advancements in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. By conducting cross-modal feature fusion in the visual feature encoding stage, we can leverage the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results are readily harvested with a light-weight mask predictor. Without bells and whistles, our method surpasses the previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref by large margins.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

yz93/lavt-ris officialmentioned in paperpytorchGPL-3.0 report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Decoder Generalized Referring Expression Segmentation Image Segmentation Referring Expression Referring Expression Segmentation Semantic Segmentation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Generalized Referring Expression Segmentation	gRefCOCO	LAVT	cIoU	57.64	#9 of 13	Archive leaderboard	report
Generalized Referring Expression Segmentation	gRefCOCO	LAVT	gIoU	58.40	#9 of 13	Archive leaderboard	report
Referring Expression Segmentation	RefCOCO+ test B	LAVT	Overall IoU	55.1	#19 of 30	Archive leaderboard	report
Referring Expression Segmentation	RefCOCO+ testA	LAVT	Overall IoU	68.38	#19 of 30	Archive leaderboard	report
Referring Expression Segmentation	RefCOCO+ val	LAVT	Overall IoU	62.14	#24 of 33	Archive leaderboard	report
Referring Expression Segmentation	RefCOCOg-test	LAVT (Swin-B)	Overall IoU	62.09	#15 of 18	Archive leaderboard	report
Referring Expression Segmentation	RefCOCOg-val	LAVT	Overall IoU	61.24	#19 of 23	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer Vision Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections