{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/handsformer-keypoint-transformer-for","title":"Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation","arxiv_id":"2104.14639","date":"2021-04-29","proceeding":"CVPR 2022 1","authors":["Shreyas Hampali","Sayan Deb Sarkar","Mahdi Rad","Vincent Lepetit"],"abstract":"We propose a robust and accurate method for estimating the 3D poses of two hands in close interaction from a single color image. This is a very challenging problem, as large occlusions and many confusions between the joints may happen. State-of-the-art methods solve this problem by regressing a heatmap for each joint, which requires solving two problems simultaneously: localizing the joints and recognizing them. In this work, we propose to separate these tasks by relying on a CNN to first localize joints as 2D keypoints, and on self-attention between the CNN features at these keypoints to associate them with the corresponding hand joint. The resulting architecture, which we call \"Keypoint Transformer\", is highly efficient as it achieves state-of-the-art performance with roughly half the number of model parameters on the InterHand2.6M dataset. We also show it can be easily extended to estimate the 3D pose of an object manipulated by one or two hands with high performance. Moreover, we created a new dataset of more than 75,000 images of two hands manipulating an object fully annotated in 3D and will make it publicly available.","url_abs":"https://arxiv.org/abs/2104.14639v2","url_pdf":"https://arxiv.org/pdf/2104.14639v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"handsformer-keypoint-transformer-for","repo_url":"https://github.com/shreyashampali/kypt_transformer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-hand-pose-estimation","task_name":"3D Hand Pose Estimation"},{"task_slug":"3d-interacting-hand-pose-estimation","task_name":"3D Interacting Hand Pose Estimation"},{"task_slug":"3d-pose-estimation","task_name":"3D Pose Estimation"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"hand-object-pose","task_name":"hand-object pose"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"heatmap","method_name":"Heatmap"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-hand-pose-estimation-on-ho-3d","task":"3D Hand Pose Estimation","dataset":"HO-3D v2","model":"KPT-Transformer","rank_in_archive_order":18,"of":24,"metrics":{"AUC_J":"0.786","PA-MPJPE (mm)":"10.8"},"uses_additional_data":false},{"leaderboard":"/sota/3d-hand-pose-estimation-on-ho-3d-v3","task":"3D Hand Pose Estimation","dataset":"HO-3D v3","model":"KPT-Transformer","rank_in_archive_order":7,"of":8,"metrics":{"AUC_J":"0.785","PA-MPJPE":"10.9"},"uses_additional_data":false},{"leaderboard":"/sota/3d-interacting-hand-pose-estimation-on","task":"3D Interacting Hand Pose Estimation","dataset":"InterHand2.6M","model":"Keypoint Transformer","rank_in_archive_order":6,"of":9,"metrics":{"MPJPE Test":"12.78","MPVPE Test":"-","MRRPE Test":"29.63"},"uses_additional_data":false},{"leaderboard":"/sota/hand-object-pose-on-ho-3d","task":"hand-object pose","dataset":"HO-3D v2","model":"Keypoint-Trans","rank_in_archive_order":4,"of":9,"metrics":{"ADD-S":"21.4","Average MPJPE (mm)":"25.5","OME":"68.0","PA-MPJPE":"10.8","ST-MPJPE":"25.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.14639","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}