{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/end-to-end-human-pose-and-mesh-reconstruction","title":"End-to-End Human Pose and Mesh Reconstruction with Transformers","arxiv_id":"2012.09760","date":"2020-12-17","proceeding":"CVPR 2021 1","authors":["Kevin Lin","Lijuan Wang","Zicheng Liu"],"abstract":"We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D human pose and mesh vertices from a single image. Our method uses a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, and outputs 3D joint coordinates and mesh vertices simultaneously. Compared to existing techniques that regress pose and shape parameters, METRO does not rely on any parametric mesh models like SMPL, thus it can be easily extended to other objects such as hands. We further relax the mesh topology and allow the transformer self-attention mechanism to freely attend between any two vertices, making it possible to learn non-local relationships among mesh vertices and joints. With the proposed masked vertex modeling, our method is more robust and effective in handling challenging situations like partial occlusions. METRO generates new state-of-the-art results for human mesh reconstruction on the public Human3.6M and 3DPW datasets. Moreover, we demonstrate the generalizability of METRO to 3D hand reconstruction in the wild, outperforming existing state-of-the-art methods on FreiHAND dataset. Code and pre-trained models are available at https://github.com/microsoft/MeshTransformer.","url_abs":"https://arxiv.org/abs/2012.09760v3","url_pdf":"https://arxiv.org/pdf/2012.09760v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"end-to-end-human-pose-and-mesh-reconstruction","repo_url":"https://github.com/microsoft/MeshTransformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"3d-absolute-human-pose-estimation","task_name":"3D Absolute Human Pose Estimation"},{"task_slug":"3d-hand-pose-estimation","task_name":"3D Hand Pose Estimation"},{"task_slug":"3d-human-pose-estimation","task_name":"3D Human Pose Estimation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-hand-pose-estimation-on-dexycb","task":"3D Hand Pose Estimation","dataset":"DexYCB","model":"METRO","rank_in_archive_order":10,"of":11,"metrics":{"Average MPJPE (mm)":"15.2","MPVPE":"-","PA-MPVPE":"-","PA-VAUC":"-","Procrustes-Aligned MPJPE":"6.99","VAUC":"-"},"uses_additional_data":false},{"leaderboard":"/sota/3d-hand-pose-estimation-on-freihand","task":"3D Hand Pose Estimation","dataset":"FreiHAND","model":"METRO","rank_in_archive_order":15,"of":33,"metrics":{"PA-F@15mm":"0.984","PA-F@5mm":"0.731","PA-MPJPE":"6.5","PA-MPVPE":"6.3"},"uses_additional_data":false},{"leaderboard":"/sota/3d-hand-pose-estimation-on-hint-hand","task":"3D Hand Pose Estimation","dataset":"HInt: Hand Interactions in the wild","model":"METRO","rank_in_archive_order":8,"of":10,"metrics":{"PCK@0.05 (Ego4D) All":"13.2","PCK@0.05 (Ego4D) Occ":"8.1","PCK@0.05 (Ego4D) Visible":"15.8","PCK@0.05 (New Days) All":"14.7","PCK@0.05 (NewDays) Occ":"7.0","PCK@0.05 (NewDays) Visible":"19.2","PCK@0.05 (VISOR) All":"16.8","PCK@0.05 (VISOR) Occ":"10.2","PCK@0.05 (VISOR) Visible":"19.7"},"uses_additional_data":false},{"leaderboard":"/sota/3d-hand-pose-estimation-on-ho-3d","task":"3D Hand Pose Estimation","dataset":"HO-3D v2","model":"METRO","rank_in_archive_order":16,"of":24,"metrics":{"AUC_J":"0.792","AUC_V":"0.779","F@15mm":"0.946","F@5mm":"0.484","PA-MPJPE (mm)":"10.4","PA-MPVPE":"11.1"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2012.09760","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}