Papers › Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation
Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation
Wangbo Zhao, Kai Wang, Xiangxiang Chu, Fuzhao Xue, Xinchao Wang, Yang You
Text-based video segmentation aims to segment the target object in a video based on a describing sentence. Incorporating motion information from optical flow maps with appearance and linguistic modalities is crucial yet has been largely ignored by previous work. In this paper, we design a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation. Specifically, we propose a multi-modal video transformer, which can fuse and aggregate multi-modal and temporal features between frames. Furthermore, we design a language-guided feature fusion module to progressively fuse appearance and motion features in each feature level with guidance from linguistic features. Finally, a multi-modal alignment loss is proposed to alleviate the semantic gap between features from different modalities. Extensive experiments on A2D Sentences and J-HMDB Sentences verify the performance and the generalization ability of our method compared to the state-of-the-art methods.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | AP | 0.419 | #10 of 27 | Archive leaderboard | report |
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | IoU mean | 0.558 | #10 of 27 | Archive leaderboard | report |
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | IoU overall | 0.673 | #10 of 27 | Archive leaderboard | report |
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | Precision@0.5 | 0.645 | #10 of 27 | Archive leaderboard | report |
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | Precision@0.6 | 0.597 | #10 of 27 | Archive leaderboard | report |
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | Precision@0.7 | 0.523 | #10 of 27 | Archive leaderboard | report |
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | Precision@0.8 | 0.375 | #10 of 27 | Archive leaderboard | report |
| Referring Expression Segmentation | A2D Sentences | mmmmtbvs | Precision@0.9 | 0.13 | #10 of 27 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections