Video Captioning

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

13 leaderboard tables shown for this task, 13 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 13 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

38 datasets whose archive record lists this task, ordered by the archive's paper count. 30 shown of 38 until expanded.

Subtasks archive 2025-07-28

6 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 211 papers with code (473 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• ECO: Efficient Convolutional Network for Online Video Understanding
  24 Apr 2018 6 repositories listed Syntology ran 0 of 2 samples · 2 unverified
  In this paper, we introduce a network architecture that takes long-term content into account and enables fast per-video processing at the same time.
• Top-down Visual Saliency Guided by Captions
  21 Dec 2016 6 repositories listed
  Neural image/video captioning models can generate accurate descriptions, but their internal process of mapping regions to words is a black box and therefore difficult to explain.
• Delving Deeper into Convolutional Networks for Learning Video Representations
  19 Nov 2015 6 repositories listed
  We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs).
• Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
  1 Apr 2021 5 repositories listed Syntology ran 3 of 11 samples · 8 unverified
  Our objective in this work is video-text retrieval - in particular a joint embedding that enables efficient text-to-video retrieval.
• What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment
  8 Apr 2019 5 repositories listed
  Can performance on the task of action quality assessment (AQA) be improved by exploiting a description of the action and its quality?
• mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video
  1 Feb 2023 4 repositories listed Syntology ran 9 of 19 samples · 10 unverified
  In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal…
• Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?
  31 Dec 2022 4 repositories listed Syntology ran 14 of 25 samples · 11 unverified
  Most existing text-video retrieval methods focus on cross-modal matching between the visual content of videos and textual query sentences.
• Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations
  21 Nov 2022 4 repositories listed Syntology ran 3 of 4 samples · 1 unverified
  Most video-and-language representation learning approaches employ contrastive learning, e.
• Multi-modal Dense Video Captioning
  17 Mar 2020 4 repositories listed
  We apply automatic speech recognition (ASR) system to obtain a temporally aligned textual description of the speech (similar to subtitles) and treat it as a separate input alongside video frames and the corresponding…
• VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
  6 Apr 2019 4 repositories listed
  We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2)…
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs
  11 Jun 2024 3 repositories listed Syntology ran 7 of 17 samples · 10 unverified
  In this paper, we present the VideoLLaMA 2, a set of Video Large Language Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks.
• Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
  27 Feb 2023 3 repositories listed
  In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
  1 May 2020 3 repositories listed Syntology ran 5 of 13 samples · 8 unverified · 8 pointer-only (licence)
  We present HERO, a novel framework for large-scale video+language omni-representation learning.
• VideoBERT: A Joint Model for Video and Language Representation Learning
  3 Apr 2019 3 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube.
• Reconstruction Network for Video Captioning
  30 Mar 2018 3 repositories listed Syntology ran 0 of 4 samples · 4 unverified
  Unlike previous video captioning work mainly exploiting the cues of video contents to make a language description, we propose a reconstruction network (RecNet) with a novel encoder-decoder-reconstructor architecture,…
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
  12 Aug 2024 2 repositories listed Syntology ran 7 of 7 samples · 0 unverified
  We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of…
• Video ReCap: Recursive Captioning of Hour-Long Videos
  20 Feb 2024 2 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 1 pointer-only (licence)
  We utilize a curriculum learning training scheme to learn the hierarchical structure of videos, starting from clip-level captions describing atomic actions, then focusing on segment-level descriptions, and concluding…
• RTQ: Rethinking Video-language Understanding Based on Image-text Model
  1 Dec 2023 2 repositories listed
  Recent advancements in video-language understanding have been established on the foundation of image-text models, resulting in promising outcomes due to the shared knowledge between images and videos.
• SoccerNet 2023 Challenges Results
  12 Sep 2023 2 repositories listed
  More information on the tasks, challenges, and leaderboards are available on https://www.
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures
  27 Jul 2023 2 repositories listed Syntology ran 0 of 1 samples · 1 unverified · 1 pointer-only (licence)
  We then present a novel method, SurgVLP - Surgical Vision Language Pre-training, for multi-modal representation learning.
• CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning
  30 Jun 2023 2 repositories listed
  We present CausalVLR (Causal Visual-Linguistic Reasoning), an open-source toolbox containing a rich set of state-of-the-art causal relation discovery and causal inference methods for various visual-linguistic reasoning…
• PaLI-X: On Scaling up a Multilingual Vision and Language Model
  29 May 2023 2 repositories listed Syntology ran 6 of 7 samples · 1 unverified
  We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture.
• VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
  29 May 2023 2 repositories listed Syntology ran 15 of 42 samples · 27 unverified
  Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various…
• SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries
  10 Apr 2023 2 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 1 pointer-only (licence)
  By providing broadcasters with a tool to summarize the content of their video with the same level of engagement as a live game, our method could help satisfy the needs of the numerous fans who follow their team but…
• ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
  11 Mar 2023 2 repositories listed
  We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and believable outputs and significantly…
• MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration
  17 Apr 2022 2 repositories listed
  Altogether, MUGEN can help progress research in many tasks in multimodal understanding and generation.
• X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics
  18 Aug 2021 2 repositories listed
  Nevertheless, there has not been an open-source codebase in support of training and deploying numerous neural network models for cross-modal analytics in a unified and modular fashion.
• End-to-End Dense Video Captioning with Parallel Decoding
  17 Aug 2021 2 repositories listed Syntology ran 3 of 7 samples · 4 unverified
  Dense video captioning aims to generate multiple associated captions with their temporal locations from the video.
• Enriching Video Captions With Contextual Text
  29 Jul 2020 2 repositories listed
  Understanding video content and generating caption with context is an important and challenging task.
• A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer
  17 May 2020 2 repositories listed Syntology ran 0 of 7 samples · 7 unverified
  We show the effectiveness of the proposed model with audio and visual modalities on the dense video captioning task, yet the module is capable of digesting any two modalities in a sequence-to-sequence task.

Syntology lines on 17 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.