{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-spatio-temporal-representation-with","title":"Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks","arxiv_id":"1711.10305","date":"2017-11-28","proceeding":"ICCV 2017 10","authors":["Zhaofan Qiu","Ting Yao","Tao Mei"],"abstract":"Convolutional Neural Networks (CNN) have been regarded as a powerful class of\nmodels for image recognition problems. Nevertheless, it is not trivial when\nutilizing a CNN for learning spatio-temporal video representation. A few\nstudies have shown that performing 3D convolutions is a rewarding approach to\ncapture both spatial and temporal dimensions in videos. However, the\ndevelopment of a very deep 3D CNN from scratch results in expensive\ncomputational cost and memory demand. A valid question is why not recycle\noff-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple\nvariants of bottleneck building blocks in a residual learning framework by\nsimulating $3\\times3\\times3$ convolutions with $1\\times3\\times3$ convolutional\nfilters on spatial domain (equivalent to 2D CNN) plus $3\\times1\\times1$\nconvolutions to construct temporal connections on adjacent feature maps in\ntime. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net\n(P3D ResNet), that exploits all the variants of blocks but composes each in\ndifferent placement of ResNet, following the philosophy that enhancing\nstructural diversity with going deep could improve the power of neural\nnetworks. Our P3D ResNet achieves clear improvements on Sports-1M video\nclassification dataset against 3D CNN and frame-based 2D CNN by 5.3% and 1.8%,\nrespectively. We further examine the generalization performance of video\nrepresentation produced by our pre-trained P3D ResNet on five different\nbenchmarks and three different tasks, demonstrating superior performances over\nseveral state-of-the-art techniques.","url_abs":"http://arxiv.org/abs/1711.10305v1","url_pdf":"http://arxiv.org/pdf/1711.10305v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-spatio-temporal-representation-with","repo_url":"https://github.com/ZhaofanQiu/pseudo-3d-residual-networks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"learning-spatio-temporal-representation-with","repo_url":"https://github.com/qijiezhao/pseudo-3d-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"philosophy","task_name":"Philosophy"},{"task_slug":"video-classification","task_name":"Video Classification"},{"task_slug":null,"task_name":"valid"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-in-videos-on-activitynet","task":"Action Recognition","dataset":"ActivityNet","model":"P3D","rank_in_archive_order":13,"of":16,"metrics":{"mAP":"78.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-sports-1m","task":"Action Recognition","dataset":"Sports-1M","model":"P3D","rank_in_archive_order":7,"of":9,"metrics":{"Clip Hit@1":"47.9","Video hit@1 ":"66.4","Video hit@5":"87.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"P3D (ImageNet + Sports1M)","rank_in_archive_order":74,"of":91,"metrics":{"3-fold Accuracy":"88.6"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.10305","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}