Papers › Moments in Time Dataset: one million videos for event understanding
Moments in Time Dataset: one million videos for event understanding
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, Aude Oliva
We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time ("opening" is "closing" in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Action Recognition | Something-Something V1 | ResNet50 I3D (Moments pretrained) | Top 1 Accuracy | 50 | #52 of 74 | Archive leaderboard | report |
| Action Recognition | Something-Something V1 | ResNet50 I3D (Kinetics pretrained) | Top 1 Accuracy | 48.6 | #60 of 74 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | Ensemble (SVM) | Top-1 (%) | 31.16 | #2 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | Ensemble (SVM) | Top-5 (%) | 57.67 | #2 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | I3D | Top-1 (%) | 29.51 | #3 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | I3D | Top-5 (%) | 56.06 | #3 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | TRN-Multiscale | Top-1 (%) | 28.27 | #4 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | TRN-Multiscale | Top-5 (%) | 53.87 | #4 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | TSN-Flow | Top-1 (%) | 15.71 | #5 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | TSN-Flow | Top-5 (%) | 34.65 | #5 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | SoundNet | Top-1 (%) | 7.60 | #6 of 6 | Archive leaderboard | report |
| Multimodal Activity Recognition | Moments in Time Dataset | SoundNet | Top-5 (%) | 18.00 | #6 of 6 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections