{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dice-end-to-end-deformation-capture-of-hand","title":"DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image","arxiv_id":"2406.17988","date":"2024-06-26","proceeding":null,"authors":["Qingxuan Wu","Zhiyang Dou","Sirui Xu","Soshi Shimada","Chen Wang","Zhengming Yu","YuAn Liu","Cheng Lin","Zeyu Cao","Taku Komura","Vladislav Golyanik","Christian Theobalt","Wenping Wang","Lingjie Liu"],"abstract":"Reconstructing 3D hand-face interactions with deformations from a single image is a challenging yet crucial task with broad applications in AR, VR, and gaming. The challenges stem from self-occlusions during single-view hand-face interactions, diverse spatial relationships between hands and face, complex deformations, and the ambiguity of the single-view setting. The first and only method for hand-face interaction recovery, Decaf, introduces a global fitting optimization guided by contact and deformation estimation networks trained on studio-collected data with 3D annotations. However, Decaf suffers from a time-consuming optimization process and limited generalization capability due to its reliance on 3D annotations of hand-face interaction data. To address these issues, we present DICE, the first end-to-end method for Deformation-aware hand-face Interaction reCovEry from a single image. DICE estimates the poses of hands and faces, contacts, and deformations simultaneously using a Transformer-based architecture. It features disentangling the regression of local deformation fields and global mesh vertex locations into two network branches, enhancing deformation and contact estimation for precise and robust hand-face mesh recovery. To improve generalizability, we propose a weakly-supervised training approach that augments the training set using in-the-wild images without 3D ground-truth annotations, employing the depths of 2D keypoints estimated by off-the-shelf models and adversarial priors of poses for supervision. Our experiments demonstrate that DICE achieves state-of-the-art performance on a standard benchmark and in-the-wild data in terms of accuracy and physical plausibility. Additionally, our method operates at an interactive rate (20 fps) on an Nvidia 4090 GPU, whereas Decaf requires more than 15 seconds for a single image. Our code will be publicly available upon publication.","url_abs":"https://arxiv.org/abs/2406.17988v1","url_pdf":"https://arxiv.org/pdf/2406.17988v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.17988","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.17988"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Qingxuan-Wu/DICE","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"09d05c2146da25a2","entry":"FCResBlock","repo":"Qingxuan-Wu/DICE","repo_kind":"found_in_text","path":"src/modeling/bert/model.py","file_url":"https://github.com/Qingxuan-Wu/DICE/blob/HEAD/src/modeling/bert/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09d05c2146da25a2"}},{"code_sha256_prefix":"493cb72fad8fd8cd","entry":"Face_Parameter_Regressor","repo":"Qingxuan-Wu/DICE","repo_kind":"found_in_text","path":"src/modeling/bert/model.py","file_url":"https://github.com/Qingxuan-Wu/DICE/blob/HEAD/src/modeling/bert/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"493cb72fad8fd8cd"}},{"code_sha256_prefix":"ef0652af6418018c","entry":"Hand_Parameter_Regressor","repo":"Qingxuan-Wu/DICE","repo_kind":"found_in_text","path":"src/modeling/bert/model.py","file_url":"https://github.com/Qingxuan-Wu/DICE/blob/HEAD/src/modeling/bert/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef0652af6418018c"}},{"code_sha256_prefix":"7dcec7ed249df88e","entry":"contains_nan","repo":"Qingxuan-Wu/DICE","repo_kind":"found_in_text","path":"src/modeling/bert/model.py","file_url":"https://github.com/Qingxuan-Wu/DICE/blob/HEAD/src/modeling/bert/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7dcec7ed249df88e"}},{"code_sha256_prefix":"fab9bd8cf44d2f5d","entry":"DICE_Network","repo":"Qingxuan-Wu/DICE","repo_kind":"found_in_text","path":"src/modeling/bert/model.py","file_url":"https://github.com/Qingxuan-Wu/DICE/blob/HEAD/src/modeling/bert/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fab9bd8cf44d2f5d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}