{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-video-representation-learning-5","title":"Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics","arxiv_id":"2008.13426","date":"2020-08-31","proceeding":null,"authors":["Jiangliu Wang","Jianbo Jiao","Linchao Bao","Shengfeng He","Wei Liu","Yun-hui Liu"],"abstract":"This paper proposes a novel pretext task to address the self-supervised video representation learning problem. Specifically, given an unlabeled video clip, we compute a series of spatio-temporal statistical summaries, such as the spatial location and dominant direction of the largest motion, the spatial location and dominant color of the largest color diversity along the temporal axis, etc. Then a neural network is built and trained to yield the statistical summaries given the video frames as inputs. In order to alleviate the learning difficulty, we employ several spatial partitioning patterns to encode rough spatial locations instead of exact spatial Cartesian coordinates. Our approach is inspired by the observation that human visual system is sensitive to rapidly changing contents in the visual field, and only needs impressions about rough spatial locations to understand the visual contents. To validate the effectiveness of the proposed approach, we conduct extensive experiments with four 3D backbone networks, i.e., C3D, 3D-ResNet, R(2+1)D and S3D-G. The results show that our approach outperforms the existing approaches across these backbone networks on four downstream video analysis tasks including action recognition, video retrieval, dynamic scene recognition, and action similarity labeling. The source code is publicly available at: https://github.com/laura-wang/video_repres_sts.","url_abs":"https://arxiv.org/abs/2008.13426v2","url_pdf":"https://arxiv.org/pdf/2008.13426v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-video-representation-learning-5","repo_url":"https://github.com/laura-wang/video_repres_sts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"self-supervised-video-representation-learning-5","repo_url":"https://github.com/laura-wang/video_repres_mas","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"scene-recognition","task_name":"Scene Recognition"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[{"method_slug":"2-1-d-convolution","method_name":"(2+1)D Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"r-2-1-d","method_name":"R(2+1)D"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2008.13426","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2008.13426"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/laura-wang/video_repres_mas","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/laura-wang/video_repres_sts","reach":null}],"summary":{"ran_honours":3,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"ccf44b771f7e8ab9","entry":"cell_gradient","repo":"laura-wang/video_repres_sts","repo_kind":"official","path":"sts/compute_motion_statistics_fast.py","file_url":"https://github.com/laura-wang/video_repres_sts/blob/HEAD/sts/compute_motion_statistics_fast.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ccf44b771f7e8ab9"}},{"code_sha256_prefix":"68fda699840fc37b","entry":"compute_motion_pattern_1","repo":"laura-wang/video_repres_sts","repo_kind":"official","path":"sts/compute_motion_statistics_fast.py","file_url":"https://github.com/laura-wang/video_repres_sts/blob/HEAD/sts/compute_motion_statistics_fast.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68fda699840fc37b"}},{"code_sha256_prefix":"68394755cc7801db","entry":"compute_motion_pattern_2","repo":"laura-wang/video_repres_sts","repo_kind":"official","path":"sts/compute_motion_statistics_fast.py","file_url":"https://github.com/laura-wang/video_repres_sts/blob/HEAD/sts/compute_motion_statistics_fast.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68394755cc7801db"}},{"code_sha256_prefix":"7b838307bdea4c62","entry":"pattern_1","repo":"laura-wang/video_repres_mas","repo_kind":"listed","path":"comput_motion_statistics_fast.py","file_url":"https://github.com/laura-wang/video_repres_mas/blob/HEAD/comput_motion_statistics_fast.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b838307bdea4c62"}},{"code_sha256_prefix":"de8e1dcb0336aaff","entry":"pattern_2","repo":"laura-wang/video_repres_mas","repo_kind":"listed","path":"comput_motion_statistics_fast.py","file_url":"https://github.com/laura-wang/video_repres_mas/blob/HEAD/comput_motion_statistics_fast.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de8e1dcb0336aaff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}