{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-spatiotemporal-feature-learning","title":"Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification","arxiv_id":"1712.04851","date":"2017-12-13","proceeding":"ECCV 2018 9","authors":["Saining Xie","Chen Sun","Jonathan Huang","Zhuowen Tu","Kevin Murphy"],"abstract":"Despite the steady progress in video analysis led by the adoption of\nconvolutional neural networks (CNNs), the relative improvement has been less\ndrastic as that in 2D static image classification. Three main challenges exist\nincluding spatial (image) feature representation, temporal information\nrepresentation, and model/computation complexity. It was recently shown by\nCarreira and Zisserman that 3D CNNs, inflated from 2D networks and pretrained\non ImageNet, could be a promising way for spatial and temporal representation\nlearning. However, as for model/computation complexity, 3D CNNs are much more\nexpensive than 2D CNNs and prone to overfit. We seek a balance between speed\nand accuracy by building an effective and efficient video classification system\nthrough systematic exploration of critical network design choices. In\nparticular, we show that it is possible to replace many of the 3D convolutions\nby low-cost 2D convolutions. Rather surprisingly, best result (in both speed\nand accuracy) is achieved when replacing the 3D convolutions at the bottom of\nthe network, suggesting that temporal representation learning on high-level\nsemantic features is more useful. Our conclusion generalizes to datasets with\nvery different properties. When combined with several other cost-effective\ndesigns including separable spatial/temporal convolution and feature gating,\nour system results in an effective video classification system that that\nproduces very competitive results on several action classification benchmarks\n(Kinetics, Something-something, UCF101 and HMDB), as well as two action\ndetection (localization) benchmarks (JHMDB and UCF101-24).","url_abs":"http://arxiv.org/abs/1712.04851v2","url_pdf":"http://arxiv.org/pdf/1712.04851v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-spatiotemporal-feature-learning","repo_url":"https://github.com/3dperceptionlab/visual-wetlandbirds","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"rethinking-spatiotemporal-feature-learning","repo_url":"https://github.com/kylemin/S3D","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"video-classification","task_name":"Video Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"S3D-G (RGB+Flow, ImageNet pretrained)","rank_in_archive_order":144,"of":207,"metrics":{"Acc@1":"77.2","Acc@5":"93"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"S3D-G (RGB, ImageNet pretrained)","rank_in_archive_order":160,"of":207,"metrics":{"Acc@1":"74.7","Acc@5":"93.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"S3D-G (Flow, ImageNet pretrained)","rank_in_archive_order":184,"of":207,"metrics":{"Acc@1":"68","Acc@5":"87.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"S3D-G (RGB+Flow)","rank_in_archive_order":55,"of":65,"metrics":{"Top-1 Accuracy":"78.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"S3D-G (RGB)","rank_in_archive_order":58,"of":65,"metrics":{"Top-1 Accuracy":"76.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"S3D-G (Flow)","rank_in_archive_order":63,"of":65,"metrics":{"Top-1 Accuracy":"69.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-hmdb-51","task":"Action Recognition","dataset":"HMDB-51","model":"S3D-G (ImageNet, Kinetics-400 pretrained)","rank_in_archive_order":37,"of":77,"metrics":{"Average accuracy of 3 splits":"75.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"S3D-G (ImageNet pretrained)","rank_in_archive_order":62,"of":74,"metrics":{"Top 1 Accuracy":"48.2","Top 5 Accuracy":"78.7"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"S3D","rank_in_archive_order":64,"of":74,"metrics":{"Top 1 Accuracy":"47.3","Top 5 Accuracy":"78.1"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"S3D-G (ImageNet, Kinetics-400 pretrained)","rank_in_archive_order":31,"of":91,"metrics":{"3-fold Accuracy":"96.8"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1712.04851","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}