{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/in-my-perspective-in-my-hands-accurate","title":"In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition","arxiv_id":"2404.09308","date":"2024-04-14","proceeding":null,"authors":["Wiktor Mucha","Martin Kampel"],"abstract":"Action recognition is essential for egocentric video understanding, allowing automatic and continuous monitoring of Activities of Daily Living (ADLs) without user effort. Existing literature focuses on 3D hand pose input, which requires computationally intensive depth estimation networks or wearing an uncomfortable depth sensor. In contrast, there has been insufficient research in understanding 2D hand pose for egocentric action recognition, despite the availability of user-friendly smart glasses in the market capable of capturing a single RGB image. Our study aims to fill this research gap by exploring the field of 2D hand pose estimation for egocentric action recognition, making two contributions. Firstly, we introduce two novel approaches for 2D hand pose estimation, namely EffHandNet for single-hand estimation and EffHandEgoNet, tailored for an egocentric perspective, capturing interactions between hands and objects. Both methods outperform state-of-the-art models on H2O and FPHA public benchmarks. Secondly, we present a robust action recognition architecture from 2D hand and object poses. This method incorporates EffHandEgoNet, and a transformer-based action recognition method. Evaluated on H2O and FPHA datasets, our architecture has a faster inference time and achieves an accuracy of 91.32% and 94.43%, respectively, surpassing state of the art, including 3D-based methods. Our work demonstrates that using 2D skeletal data is a robust approach for egocentric action understanding. Extensive evaluation and ablation studies show the impact of the hand pose estimation approach, and how each input affects the overall performance.","url_abs":"https://arxiv.org/abs/2404.09308v2","url_pdf":"https://arxiv.org/pdf/2404.09308v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"in-my-perspective-in-my-hands-accurate","repo_url":"https://github.com/wiktormucha/effhandegonet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"hand-pose-estimation","task_name":"Hand Pose Estimation"},{"task_slug":"skeleton-based-action-recognition","task_name":"Skeleton Based Action Recognition"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-on-h2o-2-hands-and-objects","task":"Action Recognition","dataset":"H2O  (2 Hands and Objects)","model":"EffHandEgoNet","rank_in_archive_order":3,"of":11,"metrics":{"Actions Top-1":"91.32","Hand Pose":"2D","Object Label":"Yes","Object Pose":"2D","RGB":"No"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-first","task":"Skeleton Based Action Recognition","dataset":"First-Person Hand Action Benchmark","model":"EffHandEgoNet","rank_in_archive_order":3,"of":4,"metrics":{"1:1 Accuracy":"94.43"},"uses_additional_data":false},{"leaderboard":"/sota/skeleton-based-action-recognition-on-h2o-2","task":"Skeleton Based Action Recognition","dataset":"H2O  (2 Hands and Objects)","model":"EffHandEgoNet","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"91.32"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.09308","atlas_url":"https://app.syntology.ai/?focus=2404.09308","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.09308"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wiktormucha/effhandegonet","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"5e9b8e9c166b9ac0","entry":"cam_2_image","repo":"wiktormucha/effhandegonet","repo_kind":"official","path":"datasets/FPHA.py","file_url":"https://github.com/wiktormucha/effhandegonet/blob/HEAD/datasets/FPHA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e9b8e9c166b9ac0"}},{"code_sha256_prefix":"db751aacfda37b09","entry":"get_positional_embeddings","repo":"wiktormucha/effhandegonet","repo_kind":"official","path":"models/models.py","file_url":"https://github.com/wiktormucha/effhandegonet/blob/HEAD/models/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db751aacfda37b09"}},{"code_sha256_prefix":"b147cbf374b4d952","entry":"vanish_keypoints","repo":"wiktormucha/effhandegonet","repo_kind":"official","path":"datasets/h2o.py","file_url":"https://github.com/wiktormucha/effhandegonet/blob/HEAD/datasets/h2o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b147cbf374b4d952"}},{"code_sha256_prefix":"23b958f13f82c222","entry":"wrlds_2_cam","repo":"wiktormucha/effhandegonet","repo_kind":"official","path":"datasets/FPHA.py","file_url":"https://github.com/wiktormucha/effhandegonet/blob/HEAD/datasets/FPHA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23b958f13f82c222"}},{"code_sha256_prefix":"ff13c0e8bc7c258c","entry":"albumentation_to_sequence","repo":"wiktormucha/effhandegonet","repo_kind":"official","path":"datasets/common_data_utils.py","file_url":"https://github.com/wiktormucha/effhandegonet/blob/HEAD/datasets/common_data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff13c0e8bc7c258c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}