{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/egoexolearn-a-dataset-for-bridging","title":"EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World","arxiv_id":"2403.16182","date":"2024-03-24","proceeding":"CVPR 2024 1","authors":["Yifei HUANG","Guo Chen","Jilan Xu","Mingfang Zhang","Lijin Yang","Baoqi Pei","Hongjie Zhang","Lu Dong","Yali Wang","LiMin Wang","Yu Qiao"],"abstract":"Being able to map the activities of others into one's own point of view is one fundamental human skill even from a very early age. Taking a step toward understanding this human ability, we introduce EgoExoLearn, a large-scale dataset that emulates the human demonstration following process, in which individuals record egocentric videos as they execute tasks guided by demonstration videos. Focusing on the potential applications in daily assistance and professional support, EgoExoLearn contains egocentric and demonstration video data spanning 120 hours captured in daily life scenarios and specialized laboratories. Along with the videos we record high-quality gaze data and provide detailed multimodal annotations, formulating a playground for modeling the human ability to bridge asynchronous procedural actions from different viewpoints. To this end, we present benchmarks such as cross-view association, cross-view action planning, and cross-view referenced skill assessment, along with detailed analysis. We expect EgoExoLearn can serve as an important resource for bridging the actions across views, thus paving the way for creating AI agents capable of seamlessly learning by observing humans in the real world. Code and data can be found at: https://github.com/OpenGVLab/EgoExoLearn","url_abs":"https://arxiv.org/abs/2403.16182v2","url_pdf":"https://arxiv.org/pdf/2403.16182v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"egoexolearn-a-dataset-for-bridging","repo_url":"https://github.com/opengvlab/egoexolearn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-anticipation","task_name":"Action Anticipation"},{"task_slug":"action-quality-assessment","task_name":"Action Quality Assessment"},{"task_slug":"long-term-anticipation","task_name":"Long Term Anticipation"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"egoexolearn","name":"EgoExoLearn","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-anticipation-on-egoexolearn","task":"Action Anticipation","dataset":"EgoExoLearn","model":"Action anticipation baseline (co-training, with gaze)","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"45.45"},"uses_additional_data":true},{"leaderboard":"/sota/action-anticipation-on-egoexolearn","task":"Action Anticipation","dataset":"EgoExoLearn","model":"Action anticipation baseline (co-training, no gaze)","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"38.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-quality-assessment-on-egoexolearn","task":"Action Quality Assessment","dataset":"EgoExoLearn","model":"RAAN+TL+Gaze","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"81.27"},"uses_additional_data":true},{"leaderboard":"/sota/action-quality-assessment-on-egoexolearn","task":"Action Quality Assessment","dataset":"EgoExoLearn","model":"RAAN+TL","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"79.875"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-egoexolearn","task":"Video Retrieval","dataset":"EgoExoLearn","model":"cross-view association baseline (gaze, val)","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"48.35"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-egoexolearn","task":"Video Retrieval","dataset":"EgoExoLearn","model":"cross-view association baseline (no gaze, val)","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"44.15"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.16182","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.16182"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/egoexolearn","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/OpenGVLab/EgoExoLearn","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5dafc748f41191f5","entry":"attentive_entropy","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"action_anticipation_planning_benchmark/loss.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/action_anticipation_planning_benchmark/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dafc748f41191f5"}},{"code_sha256_prefix":"829b3a3e9d894fa8","entry":"attentive_entropy","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"temporal_action_segmentation_benchmark/loss.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/temporal_action_segmentation_benchmark/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"829b3a3e9d894fa8"}},{"code_sha256_prefix":"b25ea1a523927d43","entry":"binary_precision_recall_curve","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"action_anticipation_planning_benchmark/anticipation_main.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/action_anticipation_planning_benchmark/anticipation_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b25ea1a523927d43"}},{"code_sha256_prefix":"b1e8dce608f3f658","entry":"cross_entropy_soft","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"action_anticipation_planning_benchmark/loss.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/action_anticipation_planning_benchmark/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1e8dce608f3f658"}},{"code_sha256_prefix":"0c3c2aa9cf2ac2f0","entry":"dis_MCD","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"action_anticipation_planning_benchmark/loss.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/action_anticipation_planning_benchmark/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c3c2aa9cf2ac2f0"}},{"code_sha256_prefix":"e1dc22b6c3aff0f2","entry":"dis_mcd","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"temporal_action_segmentation_benchmark/loss.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/temporal_action_segmentation_benchmark/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1dc22b6c3aff0f2"}},{"code_sha256_prefix":"2b38a33ad005fe8c","entry":"mean_average_precision","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"action_anticipation_planning_benchmark/anticipation_main.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/action_anticipation_planning_benchmark/anticipation_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b38a33ad005fe8c"}},{"code_sha256_prefix":"215d8bffab54d8ff","entry":"train","repo":"OpenGVLab/EgoExoLearn","repo_kind":"official","path":"action_anticipation_planning_benchmark/anticipation_main.py","file_url":"https://github.com/OpenGVLab/EgoExoLearn/blob/HEAD/action_anticipation_planning_benchmark/anticipation_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"215d8bffab54d8ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}