{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/relational-self-attention-what-s-missing-in","title":"Relational Self-Attention: What's Missing in Attention for Video Understanding","arxiv_id":"2111.01673","date":"2021-11-02","proceeding":"NeurIPS 2021 12","authors":["Manjin Kim","Heeseung Kwon","Chunyu Wang","Suha Kwak","Minsu Cho"],"abstract":"Convolution has been arguably the most important feature transform for modern neural networks, leading to the advance of deep learning. Recent emergence of Transformer networks, which replace convolution layers with self-attention blocks, has revealed the limitation of stationary convolution kernels and opened the door to the era of dynamic feature transforms. The existing dynamic transforms, including self-attention, however, are all limited for video understanding where correspondence relations in space and time, i.e., motion information, are crucial for effective representation. In this work, we introduce a relational feature transform, dubbed the relational self-attention (RSA), that leverages rich structures of spatio-temporal relations in videos by dynamically generating relational kernels and aggregating relational contexts. Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts, achieving the state of the art on the standard motion-centric benchmarks for video action recognition, such as Something-Something-V1 & V2, Diving48, and FineGym.","url_abs":"https://arxiv.org/abs/2111.01673v1","url_pdf":"https://arxiv.org/pdf/2111.01673v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"relational-self-attention-what-s-missing-in","repo_url":"https://github.com/KimManjin/RSA","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-on-diving-48","task":"Action Recognition","dataset":"Diving-48","model":"RSANet-R50 (16 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":12,"of":18,"metrics":{"Accuracy":"84.2"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"RSANet-R50 (8+16 frames, ImageNet pretrained, 2 clips)","rank_in_archive_order":23,"of":74,"metrics":{"Top 1 Accuracy":"56.1","Top 5 Accuracy":"82.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"RSANet-R50 (8+16 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":25,"of":74,"metrics":{"Top 1 Accuracy":"55.5","Top 5 Accuracy":"82.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"RSANet-R50 (16 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":34,"of":74,"metrics":{"Top 1 Accuracy":"54.0","Top 5 Accuracy":"81.1"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"RSANet-R50 (8 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":46,"of":74,"metrics":{"Top 1 Accuracy":"51.9","Top 5 Accuracy":"79.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"RSANet-R50 (8+16 frames, ImageNet pretrained, 2 clips","rank_in_archive_order":62,"of":123,"metrics":{"Top-1 Accuracy":"67.7","Top-5 Accuracy":"91.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"RSANet-R50 (8+16 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":67,"of":123,"metrics":{"Top-1 Accuracy":"67.3","Top-5 Accuracy":"90.8"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"RSANet-R50 (16 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":83,"of":123,"metrics":{"Top-1 Accuracy":"66","Top-5 Accuracy":"89.8"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"RSANet-R50 (8 frames, ImageNet pretrained, a single clip)","rank_in_archive_order":92,"of":123,"metrics":{"Top-1 Accuracy":"64.8","Top-5 Accuracy":"89.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"RSANet-R50 (8+16 frames, ImageNet pretrained, 2 clips)","rank_in_archive_order":121,"of":123,"metrics":{"Top-5 Accuracy":"91.1"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.01673","atlas_url":"https://app.syntology.ai/?focus=2111.01673","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.01673"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KimManjin/RSA","reach":null}],"summary":{"ran":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"17dc37e26f9a98c6","entry":"RSA","repo":"KimManjin/RSA","repo_kind":"official","path":"ops/rsa.py","file_url":"https://github.com/KimManjin/RSA/blob/HEAD/ops/rsa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17dc37e26f9a98c6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}