Papers › Eidetic 3D LSTM: A Model for Video Prediction and Beyond

Eidetic 3D LSTM: A Model for Video Prediction and Beyond

1 May 2019ICLR 2019 5archive 2025-07-28

Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, Li Fei-Fei

Spatiotemporal predictive learning, though long considered to be a promising self-supervised feature learning method, seldom shows its effectiveness beyond future video prediction. The reason is that it is difficult to learn good representations for both short-term frame dependency and long-term high-level relations. We present a new model, Eidetic 3D LSTM (E3D-LSTM), that integrates 3D convolutions into RNNs. The encapsulated 3D-Conv makes local perceptrons of RNNs motion-aware and enables the memory cell to store better short-term features. For long-term relations, we make the present memory state interact with its historical records via a gate-controlled self-attention module. We describe this memory transition mechanism eidetic as it is able to effectively recall the stored memories across multiple time stamps even after long periods of disturbance. We first evaluate the E3D-LSTM network on widely-used future video prediction datasets and achieve the state-of-the-art performance. Then we show that the E3D-LSTM network also performs well on the early activity recognition to infer what is happening or what will happen after observing only limited frames of video. This task aligns well with video prediction, as action intentions and tendency are important to achieve the state-of-the-art performance.

Paper PDF Code

Code

chengtan9907/simvpv2 pytorchApache-2.0 report

google/e3d_lstm tf report

metrofun/E3D-LSTM pytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Activity Recognition Video Prediction Weather Forecasting

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Video Prediction	Human3.6M	E3D-LSTM	MAE	1660	#6 of 9	Archive leaderboard	report
Video Prediction	Human3.6M	E3D-LSTM	MSE	464	#6 of 9	Archive leaderboard	report
Video Prediction	Human3.6M	E3D-LSTM	SSIM	0.869	#6 of 9	Archive leaderboard	report
Video Prediction	KTH	E3d-LSTM	Cond	10	#15 of 31	Archive leaderboard	report
Video Prediction	KTH	E3d-LSTM	PSNR	29.31	#15 of 31	Archive leaderboard	report
Video Prediction	KTH	E3d-LSTM	Pred	20	#15 of 31	Archive leaderboard	report
Video Prediction	KTH	E3d-LSTM	SSIM	0.879	#15 of 31	Archive leaderboard	report
Video Prediction	Moving MNIST	E3D-LSTM	MAE	86.4	#23 of 31	Archive leaderboard	report
Video Prediction	Moving MNIST	E3D-LSTM	MSE	41.3	#23 of 31	Archive leaderboard	report
Video Prediction	Moving MNIST	E3D-LSTM	SSIM	0.910	#23 of 31	Archive leaderboard	report
Weather Forecasting	SEVIR	E3D-LSTM	MSE	4.1702	#7 of 8	Archive leaderboard	report
Weather Forecasting	SEVIR	E3D-LSTM	mCSI	0.4038	#7 of 8	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

LSTM Sigmoid Activation Tanh Activation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections