{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/egolife-towards-egocentric-life-assistant","title":"EgoLife: Towards Egocentric Life Assistant","arxiv_id":"2503.03803","date":"2025-03-05","proceeding":"CVPR 2025 1","authors":["Jingkang Yang","Shuai Liu","Hongming Guo","Yuhao Dong","Xiamengwei Zhang","Sicheng Zhang","Pengyun Wang","Zitang Zhou","Binzhu Xie","Ziyue Wang","Bei Ouyang","Zhengyu Lin","Marco Cominelli","Zhongang Cai","Yuanhan Zhang","Peiyuan Zhang","Fangzhou Hong","Joerg Widmer","Francesco Gringoli","Lei Yang","Bo Li","Ziwei Liu"],"abstract":"We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities - including discussions, shopping, cooking, socializing, and entertainment - using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations. To address the key technical challenges of (1) developing robust visual-audio models for egocentric data, (2) enabling identity recognition, and (3) facilitating long-context question answering over extensive temporal information, we introduce EgoButler, an integrated system comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.","url_abs":"https://arxiv.org/abs/2503.03803v1","url_pdf":"https://arxiv.org/pdf/2503.03803v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"egolife-towards-egocentric-life-assistant","repo_url":"https://github.com/evolvinglmms-lab/egolife","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.03803","atlas_url":"https://app.syntology.ai/?focus=2503.03803","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.03803"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/evolvinglmms-lab/egolife","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":7},"by_repo_kind":{"official":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"6ac637835afcffbf","entry":"build_vision_projector","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6ac637835afcffbf"}},{"code_sha256_prefix":"8eddcc1d7a212099","entry":"build_vision_resampler","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8eddcc1d7a212099"}},{"code_sha256_prefix":"0c19a8693915a235","entry":"EgoGPTConfig","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0c19a8693915a235"}},{"code_sha256_prefix":"23931cef3f7aa97b","entry":"EgoGPTLlamaForCausalLM","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"23931cef3f7aa97b"}},{"code_sha256_prefix":"1ab656b0182ec561","entry":"EgoGPTLlamaModel","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1ab656b0182ec561"}},{"code_sha256_prefix":"140973675201831e","entry":"EgoGPTMetaForCausalLM","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"140973675201831e"}},{"code_sha256_prefix":"1b96d8f67fcbd8fc","entry":"EgoGPTMetaModel","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1b96d8f67fcbd8fc"}},{"code_sha256_prefix":"a1a495c721571af1","entry":"build_speech_encoder","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a1a495c721571af1"}},{"code_sha256_prefix":"700100fc2116c32c","entry":"build_vision_tower","repo":"evolvinglmms-lab/egolife","repo_kind":"official","path":"EgoGPT/egogpt/model/language_model/egogpt_llama.py","file_url":"https://github.com/evolvinglmms-lab/egolife/blob/HEAD/EgoGPT/egogpt/model/language_model/egogpt_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"700100fc2116c32c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}