{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/attend-to-who-you-are-supervising-self-1","title":"Attend to Who You Are: Supervising Self-Attention for Keypoint Detection and Instance-Aware Association","arxiv_id":"2111.12892","date":"2021-11-25","proceeding":null,"authors":["Sen yang","Zhicheng Wang","Ze Chen","YanJie Li","Shoukui Zhang","Zhibin Quan","Shu-Tao Xia","Yiping Bao","Erjin Zhou","Wankou Yang"],"abstract":"This paper presents a new method to solve keypoint detection and instance association by using Transformer. For bottom-up multi-person pose estimation models, they need to detect keypoints and learn associative information between keypoints. We argue that these problems can be entirely solved by Transformer. Specifically, the self-attention in Transformer measures dependencies between any pair of locations, which can provide association information for keypoints grouping. However, the naive attention patterns are still not subjectively controlled, so there is no guarantee that the keypoints will always attend to the instances to which they belong. To address it we propose a novel approach of supervising self-attention for multi-person keypoint detection and instance association. By using instance masks to supervise self-attention to be instance-aware, we can assign the detected keypoints to their corresponding instances based on the pairwise attention scores, without using pre-defined offset vector fields or embedding like CNN-based bottom-up models. An additional benefit of our method is that the instance segmentation results of any number of people can be directly obtained from the supervised attention matrix, thereby simplifying the pixel assignment pipeline. The experiments on the COCO multi-person keypoint detection challenge and person instance segmentation task demonstrate the effectiveness and simplicity of the proposed method and show a promising way to control self-attention behavior for specific purposes.","url_abs":"https://arxiv.org/abs/2111.12892v1","url_pdf":"https://arxiv.org/pdf/2111.12892v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"attend-to-who-you-are-supervising-self-1","repo_url":"https://github.com/yangsenius/ssa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"exists_but_no_tree"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"keypoint-detection","task_name":"Keypoint Detection"},{"task_slug":"multi-person-pose-estimation","task_name":"Multi-Person Pose Estimation"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/multi-person-pose-estimation-on-coco","task":"Multi-Person Pose Estimation","dataset":"COCO (Common Objects in Context)","model":"Supervising Self-Attention","rank_in_archive_order":10,"of":15,"metrics":{"AP":"0.665"},"uses_additional_data":false},{"leaderboard":"/sota/multi-person-pose-estimation-on-coco-test-dev","task":"Multi-Person Pose Estimation","dataset":"COCO test-dev","model":"Supervising Self-Attention","rank_in_archive_order":11,"of":15,"metrics":{"AP":"66.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.12892","atlas_url":"https://app.syntology.ai/?focus=2111.12892","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}