{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/d3d-distilled-3d-networks-for-video-action","title":"D3D: Distilled 3D Networks for Video Action Recognition","arxiv_id":"1812.08249","date":"2018-12-19","proceeding":null,"authors":["Jonathan C. Stroud","David A. Ross","Chen Sun","Jia Deng","Rahul Sukthankar"],"abstract":"State-of-the-art methods for video action recognition commonly use an\nensemble of two networks: the spatial stream, which takes RGB frames as input,\nand the temporal stream, which takes optical flow as input. In recent work,\nboth of these streams consist of 3D Convolutional Neural Networks, which apply\nspatiotemporal filters to the video clip before performing classification.\nConceptually, the temporal filters should allow the spatial stream to learn\nmotion representations, making the temporal stream redundant. However, we still\nsee significant benefits in action recognition performance by including an\nentirely separate temporal stream, indicating that the spatial stream is\n\"missing\" some of the signal captured by the temporal stream. In this work, we\nfirst investigate whether motion representations are indeed missing in the\nspatial stream of 3D CNNs. Second, we demonstrate that these motion\nrepresentations can be improved by distillation, by tuning the spatial stream\nto predict the outputs of the temporal stream, effectively combining both\nmodels into a single stream. Finally, we show that our Distilled 3D Network\n(D3D) achieves performance on par with two-stream approaches, using only a\nsingle model and with no need to compute optical flow.","url_abs":"http://arxiv.org/abs/1812.08249v2","url_pdf":"http://arxiv.org/pdf/1812.08249v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"d3d-distilled-3d-networks-for-video-action","repo_url":"https://github.com/princeton-vl/d3dhelper","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"D3D+S3D-G (RGB + RGB)","rank_in_archive_order":147,"of":207,"metrics":{"Acc@1":"76.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"D3D (RGB)","rank_in_archive_order":153,"of":207,"metrics":{"Acc@1":"75.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"D3D+S3D-G","rank_in_archive_order":53,"of":65,"metrics":{"Top-1 Accuracy":"79.1"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"D3D","rank_in_archive_order":56,"of":65,"metrics":{"Top-1 Accuracy":"77.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-ava-v21","task":"Action Recognition","dataset":"AVA v2.1","model":"D3D (ResNet RPN, Kinetics-400 pretraining)","rank_in_archive_order":11,"of":15,"metrics":{"mAP (Val)":"23"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-hmdb-51","task":"Action Recognition","dataset":"HMDB-51","model":"D3D + D3D","rank_in_archive_order":23,"of":77,"metrics":{"Average accuracy of 3 splits":"80.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-hmdb-51","task":"Action Recognition","dataset":"HMDB-51","model":"D3D (Kinetics-600 pretraining)","rank_in_archive_order":25,"of":77,"metrics":{"Average accuracy of 3 splits":"79.3"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-hmdb-51","task":"Action Recognition","dataset":"HMDB-51","model":"D3D (Kinetics-400 pretraining)","rank_in_archive_order":29,"of":77,"metrics":{"Average accuracy of 3 splits":"78.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"D3D + D3D","rank_in_archive_order":17,"of":91,"metrics":{"3-fold Accuracy":"97.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"D3D (Kinetics-600 pretraining)","rank_in_archive_order":24,"of":91,"metrics":{"3-fold Accuracy":"97.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"D3D (Kinetics-400 pretraining)","rank_in_archive_order":26,"of":91,"metrics":{"3-fold Accuracy":"97"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/1812.08249","atlas_url":"https://app.syntology.ai/?focus=1812.08249","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}