Papers › Action knowledge for video captioning with graph neural networks

Action knowledge for video captioning with graph neural networks

16 Mar 2023Journal of King Saud University - Computer and Information Sciences 2023 3archive 2025-07-28

Willy Fitra Hendria, Vania Velda, Bahy Helmi Hartoyo Putra, Fikriansyah Adzaka, Cheol Jeong

Many existing video captioning methods capture action information in the video by exploiting features extracted from an action recognition model. However, directly using the action features without object-specific representation may not well capture the object interactions. Consequently, the generated captions may not be accurate enough in describing the action and the object in the scenes. To address this issue, we propose to incorporate the action features as the edge features in a graph neural network where the nodes represent objects, thereby capturing a finer visual representation of object-action-object relationships. Previous graph-based video captioning methods commonly relied on a pretrained object detection model to create the node representations. The object detection model, however, may miss detecting some important objects. To alleviate this problem, we further introduce a grid-based node representation where the nodes are represented by the features extracted from grids of video frames. Using this representation, the important objects in the scenes are captured more thoroughly. To avoid adding any complexity during inference, the knowledge of the proposed graph is transferred to another neural network via knowledge distillation. Our proposed method achieved state-of-the-art results on two popular video captioning datasets, i.e., MSVD and MSR-VTT, on all metrics. The code of our proposed method is available at https://github.com/Sejong-VLI/V2T-Action-Graph-JKSUCIS-2023.

Code

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Results from the paper archive 2025-07-28

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections