{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hand-image-understanding-via-deep-multi-task","title":"Hand Image Understanding via Deep Multi-Task Learning","arxiv_id":"2107.11646","date":"2021-07-24","proceeding":"ICCV 2021 10","authors":["Xiong Zhang","Hongsheng Huang","Jianchao Tan","Hongmin Xu","Cheng Yang","Guozhu Peng","Lei Wang","Ji Liu"],"abstract":"Analyzing and understanding hand information from multimedia materials like images or videos is important for many real world applications and remains active in research community. There are various works focusing on recovering hand information from single image, however, they usually solve a single task, for example, hand mask segmentation, 2D/3D hand pose estimation, or hand mesh reconstruction and perform not well in challenging scenarios. To further improve the performance of these tasks, we propose a novel Hand Image Understanding (HIU) framework to extract comprehensive information of the hand object from a single RGB image, by jointly considering the relationships between these tasks. To achieve this goal, a cascaded multi-task learning (MTL) backbone is designed to estimate the 2D heat maps, to learn the segmentation mask, and to generate the intermediate 3D information encoding, followed by a coarse-to-fine learning paradigm and a self-supervised learning strategy. Qualitative experiments demonstrate that our approach is capable of recovering reasonable mesh representations even in challenging situations. Quantitatively, our method significantly outperforms the state-of-the-art approaches on various widely-used datasets, in terms of diverse evaluation metrics.","url_abs":"https://arxiv.org/abs/2107.11646v2","url_pdf":"https://arxiv.org/pdf/2107.11646v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hand-image-understanding-via-deep-multi-task","repo_url":"https://github.com/MandyMo/HIU-DMTL","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"3d-hand-pose-estimation","task_name":"3D Hand Pose Estimation"},{"task_slug":"hand-pose-estimation","task_name":"Hand Pose Estimation"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[{"slug":"hiu-dmtl-data","name":"HIU-DMTL-Data","full_name":"Hand Image Understanding via Deep Multi-Task Learning."}],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-hand-pose-estimation-on-freihand","task":"3D Hand Pose Estimation","dataset":"FreiHAND","model":"HIU-DMTL","rank_in_archive_order":22,"of":33,"metrics":{"PA-F@15mm":"0.974","PA-F@5mm":"0.699","PA-MPJPE":"7.1","PA-MPVPE":"7.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.11646","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.11646"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/MandyMo/HIU-DMTL","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c72842844140634a","entry":"get_backbone","repo":"MandyMo/HIU-DMTL","repo_kind":"official","path":"hand_detection/model/model.py","file_url":"https://github.com/MandyMo/HIU-DMTL/blob/HEAD/hand_detection/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c72842844140634a"}},{"code_sha256_prefix":"fdce4968186f142a","entry":"Model","repo":"MandyMo/HIU-DMTL","repo_kind":"official","path":"hand_detection/model/model.py","file_url":"https://github.com/MandyMo/HIU-DMTL/blob/HEAD/hand_detection/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdce4968186f142a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}