{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unsupervised-learning-of-video","title":"Unsupervised Learning of Video Representations using LSTMs","arxiv_id":"1502.04681","date":"2015-02-16","proceeding":null,"authors":["Nitish Srivastava","Elman Mansimov","Ruslan Salakhutdinov"],"abstract":"We use multilayer Long Short Term Memory (LSTM) networks to learn\nrepresentations of video sequences. Our model uses an encoder LSTM to map an\ninput sequence into a fixed length representation. This representation is\ndecoded using single or multiple decoder LSTMs to perform different tasks, such\nas reconstructing the input sequence, or predicting the future sequence. We\nexperiment with two kinds of input sequences - patches of image pixels and\nhigh-level representations (\"percepts\") of video frames extracted using a\npretrained convolutional net. We explore different design choices such as\nwhether the decoder LSTMs should condition on the generated output. We analyze\nthe outputs of the model qualitatively to see how well the model can\nextrapolate the learned video representation into the future and into the past.\nWe try to visualize and interpret the learned features. We stress test the\nmodel by running it on longer time scales and on out-of-domain data. We further\nevaluate the representations by finetuning them for a supervised learning\nproblem - human action recognition on the UCF-101 and HMDB-51 datasets. We show\nthat the representations help improve classification accuracy, especially when\nthere are only a few training examples. Even models pretrained on unrelated\ndatasets (300 hours of YouTube videos) can help action recognition performance.","url_abs":"http://arxiv.org/abs/1502.04681v3","url_pdf":"http://arxiv.org/pdf/1502.04681v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/emansim/unsupervised-videos","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/ManjunathAdi/Seq2Seq_RNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/MrSyee/rnn_autoenc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/RoshanRane/segmentation-moving-MNIST","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/duyhlzu/GMG","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/kjm1559/ADAE_LSTM_Autoencoder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/mansimov/unsupervised-videos","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/pdemeulenaer/Time-series-prediction","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/saikasyap/Video-Embeddings","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/shuuchen/video_autoencoder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/udaydutt12/Neural-Nets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"unsupervised-learning-of-video","repo_url":"https://github.com/yumaloop/LSTMAutoEncoderOnMovingMNIST","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[{"slug":"moving-mnist","name":"Moving MNIST","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1502.04681","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1502.04681"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yumaloop/LSTMAutoEncoderOnMovingMNIST","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/emansim/unsupervised-videos","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kjm1559/ADAE_LSTM_Autoencoder","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MrSyee/rnn_autoenc","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/saikasyap/Video-Embeddings","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RoshanRane/segmentation-moving-MNIST","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shuuchen/video_autoencoder","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pdemeulenaer/Time-series-prediction","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mansimov/unsupervised-videos","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ManjunathAdi/Seq2Seq_RNN","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/udaydutt12/Neural-Nets","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/duyhlzu/GMG","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":2,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"0925b30ca046f87b","entry":"mape","repo":"ManjunathAdi/Seq2Seq_RNN","repo_kind":"listed","path":"RNN_Sequence2seq_Feature-extraction.py","file_url":"https://github.com/ManjunathAdi/Seq2Seq_RNN/blob/HEAD/RNN_Sequence2seq_Feature-extraction.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0925b30ca046f87b"}},{"code_sha256_prefix":"8090cb3d730c0032","entry":"smape","repo":"ManjunathAdi/Seq2Seq_RNN","repo_kind":"listed","path":"RNN_Sequence2seq_Feature-extraction.py","file_url":"https://github.com/ManjunathAdi/Seq2Seq_RNN/blob/HEAD/RNN_Sequence2seq_Feature-extraction.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8090cb3d730c0032"}},{"code_sha256_prefix":"4ea6e043924e8d2d","entry":"DivUp","repo":"emansim/unsupervised-videos","repo_kind":"official","path":"cudamat/cudamat_conv.py","file_url":"https://github.com/emansim/unsupervised-videos/blob/HEAD/cudamat/cudamat_conv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ea6e043924e8d2d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}