{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-self-similarity-in-space-and-time-as-1","title":"Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition","arxiv_id":"2102.07092","date":"2021-02-14","proceeding":"ICCV 2021 10","authors":["Heeseung Kwon","Manjin Kim","Suha Kwak","Minsu Cho"],"abstract":"Spatio-temporal convolution often fails to learn motion dynamics in videos and thus an effective motion representation is required for video understanding in the wild. In this paper, we propose a rich and robust motion representation based on spatio-temporal self-similarity (STSS). Given a sequence of frames, STSS represents each local region as similarities to its neighbors in space and time. By converting appearance features into relational values, it enables the learner to better recognize structural patterns in space and time. We leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it. The proposed neural block, dubbed SELFY, can be easily inserted into neural architectures and trained end-to-end without additional supervision. With a sufficient volume of the neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition. Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolution. On the standard action recognition benchmarks, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results.","url_abs":"https://arxiv.org/abs/2102.07092v3","url_pdf":"https://arxiv.org/pdf/2102.07092v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-self-similarity-in-space-and-time-as-1","repo_url":"https://github.com/arunos728/SELFY","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"SELFYNet-TSM-R50En (8+16 frames, ImageNet pretrained, 2 clips)","rank_in_archive_order":19,"of":74,"metrics":{"Top 1 Accuracy":"56.6","Top 5 Accuracy":"84.4"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"SELFYNet-TSM-R50En (8+16 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":24,"of":74,"metrics":{"Top 1 Accuracy":"55.8","Top 5 Accuracy":"83.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"SELFYNet-TSM-R50 (16 frames, ImageNet pretrained)","rank_in_archive_order":32,"of":74,"metrics":{"Top 1 Accuracy":"54.3","Top 5 Accuracy":"82.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"SELFYNet-TSM-R50En (8+16 frames, ImageNet pretrained, 2 clips)","rank_in_archive_order":61,"of":123,"metrics":{"Top-1 Accuracy":"67.7","Top-5 Accuracy":"91.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"SELFYNet-TSM-R50En (8+16 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":64,"of":123,"metrics":{"Top-1 Accuracy":"67.4","Top-5 Accuracy":"91"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"SELFYNet-TSM-R50 (16 frames, ImageNet pretrained)","rank_in_archive_order":86,"of":123,"metrics":{"Top-1 Accuracy":"65.7","Top-5 Accuracy":"89.8"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2102.07092","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.07092"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arunos728/SELFY","reach":null}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"91d0b5bcdfcb245c","entry":"STSSExtraction","repo":"arunos728/SELFY","repo_kind":"official","path":"ops/selfy.py","file_url":"https://github.com/arunos728/SELFY/blob/HEAD/ops/selfy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91d0b5bcdfcb245c"}},{"code_sha256_prefix":"81bdbb798cb084e6","entry":"STSSIntegration","repo":"arunos728/SELFY","repo_kind":"official","path":"ops/selfy.py","file_url":"https://github.com/arunos728/SELFY/blob/HEAD/ops/selfy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81bdbb798cb084e6"}},{"code_sha256_prefix":"40e019d83e78ad9a","entry":"STSSTransformation","repo":"arunos728/SELFY","repo_kind":"official","path":"ops/selfy.py","file_url":"https://github.com/arunos728/SELFY/blob/HEAD/ops/selfy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40e019d83e78ad9a"}},{"code_sha256_prefix":"d00a7beec2284121","entry":"SELFYBlock","repo":"arunos728/SELFY","repo_kind":"official","path":"ops/selfy.py","file_url":"https://github.com/arunos728/SELFY/blob/HEAD/ops/selfy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d00a7beec2284121"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}