</> Code with Papers by Syntology

Papers › Actor and Action Video Segmentation from a Sentence

Actor and Action Video Segmentation from a Sentence

20 Mar 2018CVPR 2018 6arXiv:1803.07485archive 2025-07-28

Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, Cees G. M. Snoek

This paper strives for pixel-level segmentation of actors and their actions in video content. Different from existing works, which all learn to segment from a fixed vocabulary of actor and action pairs, we infer the segmentation from a natural language input sentence. This allows to distinguish between fine-grained actors in the same super-category, identify actor and action instances, and segment pairs that are outside of the actor and action vocabulary. We propose a fully-convolutional model for pixel-level actor and action segmentation using an encoder-decoder architecture optimized for video. To show the potential of actor and action video segmentation from a sentence, we extend two popular actor and action datasets with more than 7,500 natural language descriptions. Experiments demonstrate the quality of the sentence-guided segmentations, the generalization ability of our model, and its advantage for traditional actor and action segmentation compared to the state-of-the-art.

PaperPDFConference PDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

JerryX1110/awesome-rvos mentioned on GitHubMIT report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action SegmentationDecoderReferring Expression SegmentationSegmentationSentenceVideo SegmentationVideo Semantic Segmentation

Datasets

Introduced by this paper, per the archive.

A2D Sentences

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) AP 0.215 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) IoU mean 0.426 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) IoU overall 0.551 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) Precision@0.5 0.5 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) Precision@0.6 0.376 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) Precision@0.7 0.231 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) Precision@0.8 0.094 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. (Optical flow) Precision@0.9 0.004 #19 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. AP 0.198 #20 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. IoU mean 0.421 #20 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. IoU overall 0.536 #20 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. Precision@0.5 0.475 #20 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. Precision@0.6 0.347 #20 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. Precision@0.7 0.211 #20 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. Precision@0.8 0.08 #20 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences Gavriluyk el al. Precision@0.9 0.002 #20 of 27 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) AP 0.267 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) IoU mean 0.570 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) IoU overall 0.555 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) Precision@0.5 0.712 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) Precision@0.6 0.518 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) Precision@0.7 0.264 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) Precision@0.8 0.030 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. (Optical flow) Precision@0.9 0.000 #13 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. AP 0.233 #15 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. IoU mean 0.542 #15 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. IoU overall 0.541 #15 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. Precision@0.5 0.699 #15 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. Precision@0.6 0.460 #15 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. Precision@0.7 0.173 #15 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. Precision@0.8 0.014 #15 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB Gavrilyuk et al. Precision@0.9 0.000 #15 of 21 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections