{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/describing-videos-by-exploiting-temporal","title":"Describing Videos by Exploiting Temporal Structure","arxiv_id":"1502.08029","date":"2015-02-27","proceeding":"ICCV 2015 12","authors":["Li Yao","Atousa Torabi","Kyunghyun Cho","Nicolas Ballas","Christopher Pal","Hugo Larochelle","Aaron Courville"],"abstract":"Recent progress in using recurrent neural networks (RNNs) for image\ndescription has motivated the exploration of their application for video\ndescription. However, while images are static, working with videos requires\nmodeling their dynamic temporal structure and then properly integrating that\ninformation into a natural language description. In this context, we propose an\napproach that successfully takes into account both the local and global\ntemporal structure of videos to produce descriptions. First, our approach\nincorporates a spatial temporal 3-D convolutional neural network (3-D CNN)\nrepresentation of the short temporal dynamics. The 3-D CNN representation is\ntrained on video action recognition tasks, so as to produce a representation\nthat is tuned to human motion and behavior. Second we propose a temporal\nattention mechanism that allows to go beyond local temporal modeling and learns\nto automatically select the most relevant temporal segments given the\ntext-generating RNN. Our approach exceeds the current state-of-art for both\nBLEU and METEOR metrics on the Youtube2Text dataset. We also present results on\na new, larger and more challenging dataset of paired video and natural language\ndescriptions.","url_abs":"http://arxiv.org/abs/1502.08029v5","url_pdf":"http://arxiv.org/pdf/1502.08029v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"describing-videos-by-exploiting-temporal","repo_url":"https://github.com/yaoli/arctic-capgen-vid","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"describing-videos-by-exploiting-temporal","repo_url":"https://github.com/totalgood/viddesc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"describing-videos-by-exploiting-temporal","repo_url":"https://github.com/tsenghungchen/SA-tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"describing-videos-by-exploiting-temporal","repo_url":"https://github.com/hobincar/SA-LSTM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"describing-videos-by-exploiting-temporal","repo_url":"https://github.com/nasib-ullah/video-captioning-models-in-Pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":null,"task_name":"Image Description"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"video-description","task_name":"Video Description"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1502.08029","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1502.08029"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/totalgood/viddesc","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nasib-ullah/video-captioning-models-in-Pytorch","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tsenghungchen/SA-tensorflow","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaoli/arctic-capgen-vid","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hobincar/SA-LSTM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"21b15c9b43e9b7a2","entry":"entropy_loss","repo":"hobincar/SA-LSTM","repo_kind":"listed","path":"losses.py","file_url":"https://github.com/hobincar/SA-LSTM/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21b15c9b43e9b7a2"}},{"code_sha256_prefix":"68e6dbc61f0a062e","entry":"parse_batch","repo":"hobincar/SA-LSTM","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/hobincar/SA-LSTM/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68e6dbc61f0a062e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}