{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/real-time-holistic-robot-pose-estimation-with","title":"Real-time Holistic Robot Pose Estimation with Unknown States","arxiv_id":"2402.05655","date":"2024-02-08","proceeding":null,"authors":["Shikun Ban","Juling Fan","Xiaoxuan Ma","Wentao Zhu","Yu Qiao","Yizhou Wang"],"abstract":"Estimating robot pose from RGB images is a crucial problem in computer vision and robotics. While previous methods have achieved promising performance, most of them presume full knowledge of robot internal states, e.g. ground-truth robot joint angles. However, this assumption is not always valid in practical situations. In real-world applications such as multi-robot collaboration or human-robot interaction, the robot joint states might not be shared or could be unreliable. On the other hand, existing approaches that estimate robot pose without joint state priors suffer from heavy computation burdens and thus cannot support real-time applications. This work introduces an efficient framework for real-time robot pose estimation from RGB images without requiring known robot states. Our method estimates camera-to-robot rotation, robot state parameters, keypoint locations, and root depth, employing a neural network module for each task to facilitate learning and sim-to-real transfer. Notably, it achieves inference in a single feed-forward pass without iterative optimization. Our approach offers a 12-time speed increase with state-of-the-art accuracy, enabling real-time holistic robot pose estimation for the first time. Code and models are available at https://github.com/Oliverbansk/Holistic-Robot-Pose-Estimation.","url_abs":"https://arxiv.org/abs/2402.05655v4","url_pdf":"https://arxiv.org/pdf/2402.05655v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"real-time-holistic-robot-pose-estimation-with","repo_url":"https://github.com/oliverbansk/holistic-robot-pose-estimation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"real-time-holistic-robot-pose-estimation-with","repo_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"6d-pose-estimation","task_name":"6D Pose Estimation using RGB"},{"task_slug":"robot-pose-estimation","task_name":"Robot Pose Estimation"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/robot-pose-estimation-on-dream-dataset","task":"Robot Pose Estimation","dataset":"DREAM-dataset","model":"Holistic-Robot-Pose (known-joint)","rank_in_archive_order":1,"of":5,"metrics":{"AUC (avg. on 4 real DREAM datasets)":"87.7","mean-ADD (avg. on 4 real DREAM datasets)":"15.0"},"uses_additional_data":false},{"leaderboard":"/sota/robot-pose-estimation-on-dream-dataset","task":"Robot Pose Estimation","dataset":"DREAM-dataset","model":"Holistic-Robot-Pose (unknown-joint)","rank_in_archive_order":4,"of":5,"metrics":{"AUC (avg. on 4 real DREAM datasets)":"77.2","mean-ADD (avg. on 4 real DREAM datasets)":"23.1"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.05655","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05655"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/oliverbansk/holistic-robot-pose-estimation","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"90aada415cbf745a","entry":"build_frame_index","repo":"Oliverbansk/Hollistic-Robot-Pose-Estimation","repo_kind":"official","path":"lib/dataset/dream.py","file_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation/blob/HEAD/lib/dataset/dream.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90aada415cbf745a"}},{"code_sha256_prefix":"f0769b3ed47eb6e5","entry":"hnormalized","repo":"Oliverbansk/Hollistic-Robot-Pose-Estimation","repo_kind":"official","path":"lib/dataset/roboutils.py","file_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation/blob/HEAD/lib/dataset/roboutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0769b3ed47eb6e5"}},{"code_sha256_prefix":"0b1fa62fef215228","entry":"make_detections_from_segmentation","repo":"Oliverbansk/Hollistic-Robot-Pose-Estimation","repo_kind":"official","path":"lib/dataset/roboutils.py","file_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation/blob/HEAD/lib/dataset/roboutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b1fa62fef215228"}},{"code_sha256_prefix":"907da4b4e2cf6ad6","entry":"occlusion_aug","repo":"Oliverbansk/Hollistic-Robot-Pose-Estimation","repo_kind":"official","path":"lib/dataset/augmentations.py","file_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation/blob/HEAD/lib/dataset/augmentations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"907da4b4e2cf6ad6"}},{"code_sha256_prefix":"77c0370c17a68d8c","entry":"get_rootnet","repo":"Oliverbansk/Hollistic-Robot-Pose-Estimation","repo_kind":"official","path":"lib/models/depth_net.py","file_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation/blob/HEAD/lib/models/depth_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77c0370c17a68d8c"}},{"code_sha256_prefix":"67f033e7853eaca4","entry":"to_pil","repo":"Oliverbansk/Hollistic-Robot-Pose-Estimation","repo_kind":"official","path":"lib/dataset/augmentations.py","file_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation/blob/HEAD/lib/dataset/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67f033e7853eaca4"}},{"code_sha256_prefix":"825fa64d2cb85ca4","entry":"to_torch_uint8","repo":"Oliverbansk/Hollistic-Robot-Pose-Estimation","repo_kind":"official","path":"lib/dataset/augmentations.py","file_url":"https://github.com/Oliverbansk/Hollistic-Robot-Pose-Estimation/blob/HEAD/lib/dataset/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"825fa64d2cb85ca4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}