</> Code with Papers by Syntology

Papers › Segmentation from Natural Language Expressions

Segmentation from Natural Language Expressions

20 Mar 2016arXiv:1603.06180archive 2025-07-28

Ronghang Hu, Marcus Rohrbach, Trevor Darrell

In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefined set of semantic classes, as e.g., the phrase "two men sitting on the right bench" requires segmenting only the two people on the right bench and no one standing or sitting on another bench. Previous approaches suitable for this task were limited to a fixed set of categories and/or rectangular regions. To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information. In our model, a recurrent LSTM network is used to encode the referential expression into a vector representation, and a fully convolutional network is used to a extract a spatial feature map from the image and output a spatial response map for the target object. We demonstrate on a benchmark dataset that our model can produce quality segmentation output from the natural language expression, and outperforms baseline methods by a large margin.

PaperPDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

ronghanghu/text_objseg officialmentioned on GitHubtfNOASSERTION report
ssharpe42/NLQAC_Instance_Selection mentioned on GitHubtf report
ssharpe42/NLQAC_ObjSeg mentioned on GitHubtf report
ssharpe42/VNLQAC mentioned on GitHubtf report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Referring Expression SegmentationSegmentationSemantic Segmentation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Referring Expression Segmentation A2D Sentences Hu et al. AP 0.132 #22 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Hu et al. IoU mean 0.350 #22 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Hu et al. IoU overall 0.474 #22 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Hu et al. Precision@0.5 0.348 #22 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Hu et al. Precision@0.6 0.236 #22 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Hu et al. Precision@0.7 0.133 #22 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Hu et al. Precision@0.8 0.033 #22 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Hu et al. Precision@0.9 0.000 #22 of 27 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. AP 0.178 #16 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. IoU mean 0.528 #16 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. IoU overall 0.546 #16 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. Precision@0.5 0.633 #16 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. Precision@0.6 0.350 #16 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. Precision@0.7 0.085 #16 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. Precision@0.8 0.002 #16 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Hu et al. Precision@0.9 0.000 #16 of 21 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

LSTMSigmoid ActivationTanh Activation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections