{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/joint-partition-group-attention-for-skeleton","title":"Joint-Partition Group Attention for skeleton-based action recognition","arxiv_id":null,"date":"2024-07-30","proceeding":"Signal Processing 2024 7","authors":["Hu Cui","Tessai Hayama"],"abstract":"Skeleton-based action recognition aims to recognize human actions from the coordinates of human joints. By encoding coordinates as joint tokens, previous methods have successfully utilized the self-attention (SA) mechanism to capture the relationship of each pair of joints. However, the attention map generated from the joint Query and joint Key in SA only captures joint-to-joint correlations at a single granularity, which is obviously insufficient for human actions that express semantics in terms of body parts. In this paper, we argue that SA should have a more comprehensive mechanism to capture correlations in joint-to-joint and joint-to-partition patterns for a higher semantic representation of skeleton-based actions. Therefore, we propose Joint-Partition Group Attention (JPGA) to simultaneously capture correlations between joints and body parts of different granularity sizes. Specifically, JPGA integrates the joint tokens according to the joint’s human body partition attributes and produces different body parts tokens (partition-tokens) with different granularities. Then the attention map of JPGA is computed from joint-token and partition-token of different granularity sizes to represent the relationship between joints and body parts. To adaptively partition human body parts at different granularities, we apply the reparameterization trick to adaptively learn the multi-granularity partitioning matrix. Based on JPGA, we construct our Joint-Partition Former (JPFormer), and conduct extensive experiments on NTU-RGB+D, NTU-RRB+D 120, and Northwestern UCLA datasets and achieve state-of-the-art results, which highlights the effectiveness of our design and practice.","url_abs":"https://www.sciencedirect.com/science/article/pii/S0165168424002111","url_pdf":"https://www.sciencedirect.com/science/article/pii/S0165168424002111","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"joint-partition-group-attention-for-skeleton","repo_url":"https://github.com/HuCui2022/JPFormer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-in-videos-on-ntu-rgbd","task":"Action Recognition","dataset":"NTU RGB+D","model":"JPFormer","rank_in_archive_order":16,"of":28,"metrics":{"Accuracy (CS)":"93.2","Accuracy (CV)":"96.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-ntu-rgbd-120","task":"Action Recognition","dataset":"NTU RGB+D 120","model":"JPFormer(Pose)","rank_in_archive_order":11,"of":21,"metrics":{"Accuracy (Cross-Setup)":"91.4","Accuracy (Cross-Subject)":"89.4"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-n-ucla","task":"Skeleton Based Action Recognition","dataset":"N-UCLA","model":"JPFormer","rank_in_archive_order":12,"of":25,"metrics":{"Accuracy":"96.7"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-ntu-rgbd","task":"Skeleton Based Action Recognition","dataset":"NTU RGB+D","model":"JPFormer","rank_in_archive_order":17,"of":135,"metrics":{"Accuracy (CS)":"93.2","Accuracy (CV)":"96.9","Ensembled Modalities":"4"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-ntu-rgbd-1","task":"Skeleton Based Action Recognition","dataset":"NTU RGB+D 120","model":"JPFormer","rank_in_archive_order":19,"of":83,"metrics":{"Accuracy (Cross-Setup)":"91.4","Accuracy (Cross-Subject)":"89.4","Ensembled Modalities":"4"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}