Papers › Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph...
Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution
Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, Hyung Jin Chang
Estimating the pose and shape of hands and objects under interaction finds numerous applications including augmented and virtual reality. Existing approaches for hand and object reconstruction require explicitly defined physical constraints and known objects, which limits its application domains. Our algorithm is agnostic to object models, and it learns the physical rules governing hand-object interaction. This requires automatically inferring the shapes and physical interaction of hands and (potentially unknown) objects. We seek to approach this challenging problem by proposing a collaborative learning strategy where two-branches of deep networks are learning from each other. Specifically, we transfer hand mesh information to the object branch and vice versa for the hand branch. The resulting optimisation (training) problem can be unstable, and we address this via two strategies: (i) attention-guided graph convolution which helps identify and focus on mutual occlusion and (ii) unsupervised associative loss which facilitates the transfer of information between the branches. Experiments using four widely-used benchmarks show that our framework achieves beyond state-of-the-art accuracy in 3D pose estimation, as well as recovers dense 3D hand and object shapes. Each technical component above contributes meaningfully in the ablation study.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| 3D Hand Pose Estimation | HO-3D v2 | Tse et al. | F@15mm | 0.943 | #24 of 24 | Archive leaderboard | report |
| 3D Hand Pose Estimation | HO-3D v2 | Tse et al. | F@5mm | 0.485 | #24 of 24 | Archive leaderboard | report |
| 3D Hand Pose Estimation | HO-3D v2 | Tse et al. | PA-MPVPE | 10.9 | #24 of 24 | Archive leaderboard | report |
| hand-object pose | DexYCB | CLAGC | ADD-S | - | #6 of 9 | Archive leaderboard | report |
| hand-object pose | DexYCB | CLAGC | Average MPJPE (mm) | 15.3 | #6 of 9 | Archive leaderboard | report |
| hand-object pose | DexYCB | CLAGC | MCE | - | #6 of 9 | Archive leaderboard | report |
| hand-object pose | DexYCB | CLAGC | OCE | - | #6 of 9 | Archive leaderboard | report |
| hand-object pose | DexYCB | CLAGC | Procrustes-Aligned MPJPE | - | #6 of 9 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections