Video Prediction

Benchmarks archive 2025-07-28

19 leaderboard tables shown for this task, 19 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 19 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

25 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

2 subtasks in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 208 papers with code (394 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
  13 Jun 2015 23 repositories listed Syntology ran 1 of 3 samples · 2 unverified · 2 pointer-only (licence)
  The goal of precipitation nowcasting is to predict the future rainfall intensity in a local region over a relatively short period of time.
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
  25 May 2016 17 repositories listed Syntology ran 0 of 6 samples · 6 unverified · 2 pointer-only (licence)
  Here, we explore prediction of future frames in a video sequence as an unsupervised learning rule for learning about the structure of the visual world.
• PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning
  17 Apr 2018 11 repositories listed Syntology ran 0 of 7 samples · 7 unverified
  We present PredRNN++, an improved recurrent network for video predictive learning.
• Video-to-Video Synthesis
  20 Aug 2018 10 repositories listed
  We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.
• MogaNet: Multi-order Gated Aggregation Network
  7 Nov 2022 7 repositories listed Syntology ran 12 of 15 samples · 3 unverified
  Notably, MogaNet hits 80.
• Video Diffusion Models
  7 Apr 2022 5 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  Generating temporally coherent high fidelity video is an important milestone in generative modeling research.
• The "something something" video database for learning and evaluating visual common sense
  13 Jun 2017 5 repositories listed
  Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification.
• Deep multi-scale video prediction beyond mean square error
  17 Nov 2015 5 repositories listed Syntology ran 2 of 10 samples · 8 unverified
  Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics.
• Scalable Gradients for Stochastic Differential Equations
  5 Jan 2020 4 repositories listed Syntology ran 1 of 5 samples · 4 unverified · 2 pointer-only (licence)
  The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations.
• SME-Net: Sparse Motion Estimation for Parametric Video Prediction Through Reinforcement Learning
  1 Oct 2019 4 repositories listed
  Inspired by the success of sparse motion-based prediction for video compression, we propose a parametric video prediction on a sparse motion field composed of few critical pixels and their motion vectors.
• Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics
  19 Nov 2018 4 repositories listed
  Natural spatiotemporal processes can be highly non-stationary in many ways, e.
• Stochastic Adversarial Video Prediction
  4 Apr 2018 4 repositories listed Syntology ran 4 of 16 samples · 12 unverified · 3 pointer-only (licence)
  However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging -- the ambiguous nature of the problem can cause a naively designed model to average together possible futures…
• Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model
  12 Jun 2017 4 repositories listed Syntology ran 9 of 10 samples · 1 unverified · 5 pointer-only (licence)
  To address these problems, we propose both a new model and a benchmark for precipitation nowcasting.
• Learning a Driving Simulator
  3 Aug 2016 4 repositories listed Syntology ran 1 of 2 samples · 1 unverified
  Comma.
• GenAD: Generalized Predictive Model for Autonomous Driving
  14 Mar 2024 3 repositories listed Syntology ran 6 of 8 samples · 2 unverified
  In this paper, we introduce the first large-scale video prediction model in the autonomous driving discipline.
• Switch EMA: A Free Lunch for Better Flatness and Sharpness
  14 Feb 2024 3 repositories listed
  Exponential Moving Average (EMA) is a widely used weight averaging (WA) regularization to learn flat optima for better generalizations without extra cost in deep neural network (DNN) optimization.
• Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
  9 Oct 2023 3 repositories listed Syntology ran 12 of 20 samples · 8 unverified
  While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation.
• Video Prediction Models as Rewards for Reinforcement Learning
  23 May 2023 3 repositories listed Syntology ran 6 of 15 samples · 9 unverified · 1 pointer-only (licence)
  A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet.
• SimVP: Simpler yet Better Video Prediction
  9 Jun 2022 3 repositories listed Syntology ran 6 of 9 samples · 3 unverified · 9 pointer-only (licence)
  From CNN, RNN, to ViT, we have witnessed remarkable advancements in video prediction, incorporating auxiliary inputs, elaborate neural architectures, and sophisticated training strategies.
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
  17 Mar 2021 3 repositories listed
  This paper models these structures by presenting PredRNN, a new recurrent network, in which a pair of memory cells are explicitly decoupled, operate in nearly independent transition manners, and finally form unified…
• Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction
  3 Mar 2020 3 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Leveraging physical knowledge described by partial differential equations (PDEs) is an appealing way to improve unsupervised video prediction methods.
• PredNet and Predictive Coding: A Critical Review
  14 Jun 2019 3 repositories listed
  We fill in the gap by evaluating PredNet both as an implementation of the predictive coding theory and as a self-supervised video prediction model using a challenging video action classification dataset.
• Eidetic 3D LSTM: A Model for Video Prediction and Beyond
  1 May 2019 3 repositories listed
  We first evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance.
• Towards Accurate Generative Models of Video: A New Metric & Challenges
  3 Dec 2018 3 repositories listed
  To this extent we propose Fr\'{e}chet Video Distance (FVD), a new metric for generative models of video, and StarCraft 2 Videos (SCV), a benchmark of game play from custom starcraft 2 scenarios that challenge the…
• Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning
  6 Oct 2018 3 repositories listed
  We demonstrate that this idea can be combined with a video-prediction based controller to enable complex behaviors to be learned from scratch using only raw visual inputs, including grasping, repositioning objects, and…
• Stochastic Video Generation with a Learned Prior
  21 Feb 2018 3 repositories listed Syntology ran 1 of 2 samples · 1 unverified · 2 pointer-only (licence)
  Sample generations are both varied and sharp, even many frames into the future, and compare favorably to those from existing approaches.
• Stochastic Variational Video Prediction
  30 Oct 2017 3 repositories listed
  We find that our proposed method produces substantially improved video predictions when compared to the same model without stochasticity, and to other stochastic video prediction methods.
• Self-Supervised Visual Planning with Temporal Skip Connections
  15 Oct 2017 3 repositories listed
  One learning signal that is always available for autonomously collected data is prediction: if a robot can learn to predict the future, it can use this predictive model to take actions to produce desired outcomes, such…
• Video Frame Synthesis using Deep Voxel Flow
  8 Feb 2017 3 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow.
• Rethinking Urban Mobility Prediction: A Super-Multivariate Time Series Forecasting Approach
  4 Dec 2023 2 repositories listed
  To address this challenge, we present the Super-Multivariate Urban Mobility Transformer (SUMformer), which utilizes a specially designed attention mechanism to calculate temporal and cross-variable correlations and…

Syntology lines on 17 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.