{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/egovideo-exploring-egocentric-foundation","title":"EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation","arxiv_id":"2406.18070","date":"2024-06-26","proceeding":null,"authors":["Baoqi Pei","Guo Chen","Jilan Xu","Yuping He","Yicheng Liu","Kanghua Pan","Yifei HUANG","Yali Wang","Tong Lu","LiMin Wang","Yu Qiao"],"abstract":"In this report, we present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge. Building upon the video-language two-tower model and leveraging our meticulously organized egocentric video data, we introduce a novel foundation model called EgoVideo. This model is specifically designed to cater to the unique characteristics of egocentric videos and provides strong support for our competition submissions. In the Ego4D challenges, we tackle various tasks including Natural Language Queries, Step Grounding, Moment Queries, Short-term Object Interaction Anticipation, and Long-term Action Anticipation. In addition, we also participate in the EPIC-Kitchens challenge, where we engage in the Action Recognition, Multiple Instance Retrieval, and Domain Adaptation for Action Recognition tracks. By adapting EgoVideo to these diverse tasks, we showcase its versatility and effectiveness in different egocentric video analysis scenarios, demonstrating the powerful representation ability of EgoVideo as an egocentric foundation model. Our codebase and pretrained models are publicly available at https://github.com/OpenGVLab/EgoVideo.","url_abs":"https://arxiv.org/abs/2406.18070v4","url_pdf":"https://arxiv.org/pdf/2406.18070v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"egovideo-exploring-egocentric-foundation","repo_url":"https://github.com/opengvlab/egovideo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-anticipation","task_name":"Action Anticipation"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"long-term-action-anticipation","task_name":"Long Term Action Anticipation"},{"task_slug":"moment-queries","task_name":"Moment Queries"},{"task_slug":"multi-instance-retrieval","task_name":"Multi-Instance Retrieval"},{"task_slug":"natural-language-queries","task_name":"Natural Language Queries"},{"task_slug":"short-term-object-interaction-anticipation","task_name":"Short-term Object Interaction Anticipation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-queries-on-ego4d","task":"Natural Language Queries","dataset":"Ego4D","model":"EgoVideo","rank_in_archive_order":1,"of":10,"metrics":{"R@1 IoU=0.3":"28.05","R@1 IoU=0.5":"19.31","R@1 Mean(0.3 and 0.5)":"23.68","R@5 IoU=0.3":"44.16","R@5 IoU=0.5":"31.37"},"uses_additional_data":true},{"leaderboard":"/sota/short-term-object-interaction-anticipation-on","task":"Short-term Object Interaction Anticipation","dataset":"Ego4D","model":"EgoVideo","rank_in_archive_order":1,"of":4,"metrics":{"Noun (Top5 mAP)":"31.08","Noun+TTC (Top5 mAP)":"12.41","Noun+Verb(Top5 mAP)":"16.18","Overall (Top5 mAP)":"7.21"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.18070","atlas_url":"https://app.syntology.ai/?focus=2406.18070","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.18070"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/egovideo","reach":{"status":"ok"}}],"summary":{"ran":10,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":15,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"7c4c7b143a48f3a6","entry":"conv_1xnxn","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/modeling_finetune_uniformer_ego4d.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/modeling_finetune_uniformer_ego4d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c4c7b143a48f3a6"}},{"code_sha256_prefix":"3b7bc9c45c2814ab","entry":"conv_3xnxn","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/modeling_finetune_uniformer_ego4d.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/modeling_finetune_uniformer_ego4d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b7bc9c45c2814ab"}},{"code_sha256_prefix":"8d39fc7bbd28af45","entry":"conv_3xnxn_std","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/modeling_finetune_uniformer_ego4d.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/modeling_finetune_uniformer_ego4d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d39fc7bbd28af45"}},{"code_sha256_prefix":"1075afcc0bf619d6","entry":"get_1d_sincos_pos_embed","repo":"opengvlab/egovideo","repo_kind":"official","path":"backbone/model/pos_embed.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/backbone/model/pos_embed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1075afcc0bf619d6"}},{"code_sha256_prefix":"77e8a3ac46f3afec","entry":"get_2d_sincos_pos_embed","repo":"opengvlab/egovideo","repo_kind":"official","path":"backbone/model/pos_embed.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/backbone/model/pos_embed.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77e8a3ac46f3afec"}},{"code_sha256_prefix":"54f45ddeec14da87","entry":"get_3d_sincos_pos_embed","repo":"opengvlab/egovideo","repo_kind":"official","path":"backbone/model/pos_embed.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/backbone/model/pos_embed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54f45ddeec14da87"}},{"code_sha256_prefix":"9b2250a3d13ed688","entry":"get_sinusoid_encoding_table","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/modeling_finetune.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/modeling_finetune.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b2250a3d13ed688"}},{"code_sha256_prefix":"3890a5b8c4226d35","entry":"isinf","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/engine_for_finetuning_ego4d_hands.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/engine_for_finetuning_ego4d_hands.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3890a5b8c4226d35"}},{"code_sha256_prefix":"23f0053c19d0446f","entry":"isnan","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/engine_for_finetuning_ego4d_hands.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/engine_for_finetuning_ego4d_hands.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23f0053c19d0446f"}},{"code_sha256_prefix":"011d4cf8cfdc2ffb","entry":"train_class_batch","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/engine_for_finetuning_ego4d_hands.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/engine_for_finetuning_ego4d_hands.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"011d4cf8cfdc2ffb"}},{"code_sha256_prefix":"da8b1baf88d8904e","entry":"train_class_batch_verb","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/engine_for_finetuning_ego4d_sta.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/engine_for_finetuning_ego4d_sta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da8b1baf88d8904e"}},{"code_sha256_prefix":"29416043c7035e7c","entry":"get_loss_scale_for_deepspeed","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/engine_for_finetuning.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29416043c7035e7c"}},{"code_sha256_prefix":"14b3b13b2e7ee844","entry":"tensor_normalize","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/kinetics.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/kinetics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14b3b13b2e7ee844"}},{"code_sha256_prefix":"c7976ea27edc377a","entry":"train_class_batch","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/engine_for_finetuning.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7976ea27edc377a"}},{"code_sha256_prefix":"a0032907bd3ccf71","entry":"train_class_batch","repo":"opengvlab/egovideo","repo_kind":"official","path":"eccv-2022/engine_for_finetuning_ego4d_sta.py","file_url":"https://github.com/opengvlab/egovideo/blob/HEAD/eccv-2022/engine_for_finetuning_ego4d_sta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0032907bd3ccf71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}