{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-video-representation-learning","title":"Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles","arxiv_id":"1811.09795","date":"2018-11-24","proceeding":null,"authors":["Dahun Kim","Donghyeon Cho","In So Kweon"],"abstract":"Self-supervised tasks such as colorization, inpainting and zigsaw puzzle have\nbeen utilized for visual representation learning for still images, when the\nnumber of labeled images is limited or absent at all. Recently, this worthwhile\nstream of study extends to video domain where the cost of human labeling is\neven more expensive. However, the most of existing methods are still based on\n2D CNN architectures that can not directly capture spatio-temporal information\nfor video applications. In this paper, we introduce a new self-supervised task\ncalled as \\textit{Space-Time Cubic Puzzles} to train 3D CNNs using large scale\nvideo dataset. This task requires a network to arrange permuted 3D\nspatio-temporal crops. By completing \\textit{Space-Time Cubic Puzzles}, the\nnetwork learns both spatial appearance and temporal relation of video frames,\nwhich is our final goal. In experiments, we demonstrate that our learned 3D\nrepresentation is well transferred to action recognition tasks, and outperforms\nstate-of-the-art 2D CNN-based competitors on UCF101 and HMDB51 datasets.","url_abs":"http://arxiv.org/abs/1811.09795v1","url_pdf":"http://arxiv.org/pdf/1811.09795v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"colorization","task_name":"Colorization"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-action-recognition","task_name":"Self-Supervised Action Recognition"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/self-supervised-action-recognition-on-hmdb51","task":"Self-Supervised Action Recognition","dataset":"HMDB51","model":"3D Cubic Puzzles (3D ResNet-18)","rank_in_archive_order":43,"of":48,"metrics":{"Frozen":"false","Pre-Training Dataset":"Kinetics400","Top-1 Accuracy":"33.7"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-action-recognition-on-ucf101","task":"Self-Supervised Action Recognition","dataset":"UCF101","model":"3D Cubic Puzzles (3D ResNet-18)","rank_in_archive_order":42,"of":53,"metrics":{"3-fold Accuracy":"65.8","Frozen":"false","Pre-Training Dataset":"Kinetics400"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1811.09795","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}