</> Code with Papers by Syntology

Papers › Hierarchical interaction network for video object segmentation from referring expressions

Hierarchical interaction network for video object segmentation from referring expressions

22 Nov 2021British Machine Vision Conference 2021 11archive 2025-07-28

Zhao Yang, Yansong Tang, Luca Bertinetto, Hengshuang Zhao, Philip Torr

In this paper, we investigate the problem of video object segmentation from referring expressions (VOSRE). Conventional methods typically perform multi-modal fusion based on linguistic features and the visual features extracted from the top layer of the visual encoder, which limits these models' ability to represent multi-modal inputs at different semantic and spatial granularity levels. To address this issue, we present an end-to-end hierarchical interaction network (HINet) for the VOSRE problem. Our model leverages the feature pyramid produced by the visual encoder to generate multiple levels of multi-modal features. This allows more flexible representation of various linguistic concepts (e.g., object attributes and categories) in different levels of the multi-modal features. Moreover, we further extract signals of moving objects from optical flow input, and utilize them as complementary cues for highlighting the referent and suppressing the background with a motion gating mechanism. In contrast to previous methods, this strategy allows our model to make online predictions without requiring the whole video as input. Despite its simplicity, our proposed HINet improves over the previous state of the art on the DAVIS-16, DAVIS-17, and J-HMDB datasets for the VOSRE task, demonstrating its effectiveness and generality.

PaperPDF

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Optical Flow EstimationReferring Expression SegmentationSemantic SegmentationVideo Object SegmentationVideo Semantic Segmentation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Referring Expression Segmentation A2D Sentences HINet IoU mean 0.529 #23 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences HINet IoU overall 0.679 #23 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences HINet Precision@0.5 0.611 #23 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences HINet Precision@0.6 0.559 #23 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences HINet Precision@0.7 0.486 #23 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences HINet Precision@0.8 0.342 #23 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences HINet Precision@0.9 0.12 #23 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences RefVOS IoU mean 0.497 #24 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences RefVOS IoU overall 0.672 #24 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences RefVOS Precision@0.5 0.578 #24 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences RefVOS Precision@0.6 0.534 #24 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences RefVOS Precision@0.7 0.456 #24 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences RefVOS Precision@0.8 0.311 #24 of 27 Archive leaderboard report
Referring Expression Segmentation A2D Sentences RefVOS Precision@0.9 0.093 #24 of 27 Archive leaderboard report
Referring Expression Segmentation DAVIS 2017 (val) HINet J&F 1st frame 50.2 #10 of 18 Archive leaderboard report
Referring Expression Segmentation DAVIS 2017 (val) HINet J&F Full video 47.9 #10 of 18 Archive leaderboard report
Referring Expression Segmentation J-HMDB HINet IoU mean 0.627 #18 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB HINet IoU overall 0.652 #18 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB HINet Precision@0.5 0.819 #18 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB HINet Precision@0.6 0.736 #18 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB HINet Precision@0.7 0.542 #18 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB HINet Precision@0.8 0.168 #18 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB HINet Precision@0.9 0.4 #18 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB RefVOS IoU mean 0.568 #21 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB RefVOS IoU overall 0.606 #21 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB RefVOS Precision@0.5 0.731 #21 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB RefVOS Precision@0.6 0.62 #21 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB RefVOS Precision@0.7 0.392 #21 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB RefVOS Precision@0.8 0.088 #21 of 21 Archive leaderboard report
Referring Expression Segmentation J-HMDB RefVOS Precision@0.9 0.0 #21 of 21 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections