{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/videgothink-assessing-egocentric-video","title":"VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI","arxiv_id":"2410.11623","date":"2024-10-15","proceeding":null,"authors":["Sijie Cheng","Kechen Fang","Yangyang Yu","Sicheng Zhou","Bohao Li","Ye Tian","Tingguang Li","Lei Han","Yang Liu"],"abstract":"Recent advancements in Multi-modal Large Language Models (MLLMs) have opened new avenues for applications in Embodied AI. Building on previous work, EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities. To bridge the gap between MLLMs and low-level control in Embodied AI, we design four key interrelated tasks: video question-answering, hierarchy planning, visual grounding and reward modeling. To minimize manual annotation costs, we develop an automatic data generation pipeline based on the Ego4D dataset, leveraging the prior knowledge and multimodal capabilities of GPT-4o. Three human annotators then filter the generated data to ensure diversity and quality, resulting in the VidEgoThink benchmark. We conduct extensive experiments with three types of models: API-based MLLMs, open-source image-based MLLMs, and open-source video-based MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform poorly across all tasks related to egocentric video understanding. These findings suggest that foundation models still require significant advancements to be effectively applied to first-person scenarios in Embodied AI. In conclusion, VidEgoThink reflects a research trend towards employing MLLMs for egocentric vision, akin to human capabilities, enabling active observation and interaction in the complex real-world environments.","url_abs":"https://arxiv.org/abs/2410.11623v1","url_pdf":"https://arxiv.org/pdf/2410.11623v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"videgothink-assessing-egocentric-video","repo_url":"https://github.com/adacheng/egothink","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.11623","atlas_url":"https://app.syntology.ai/?focus=2410.11623","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.11623"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adacheng/egothink","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_honours":3,"ran_draft_wrong":1,"ran":8,"ran_violates":1,"unverified":2},"by_repo_kind":{"listed":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b47d48e431b34acd","entry":"apply_rotary_emb","repo":"adacheng/egothink","repo_kind":"listed","path":"models/llama_adapter_v2/llama.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/llama_adapter_v2/llama.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b47d48e431b34acd"}},{"code_sha256_prefix":"6e45201fa27cb24a","entry":"ceil_by_factor","repo":"adacheng/egothink","repo_kind":"listed","path":"models/qwen_vl_utils_new.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/qwen_vl_utils_new.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e45201fa27cb24a"}},{"code_sha256_prefix":"f41cb1a19b154297","entry":"encode_image","repo":"adacheng/egothink","repo_kind":"listed","path":"gpt_eval.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/gpt_eval.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f41cb1a19b154297"}},{"code_sha256_prefix":"8155263d7ff19bb3","entry":"floor_by_factor","repo":"adacheng/egothink","repo_kind":"listed","path":"models/qwen_vl_utils_new.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/qwen_vl_utils_new.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8155263d7ff19bb3"}},{"code_sha256_prefix":"fd48ec28ab00f691","entry":"get_generation_args","repo":"adacheng/egothink","repo_kind":"listed","path":"gpt_eval.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/gpt_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd48ec28ab00f691"}},{"code_sha256_prefix":"27d8599384686c24","entry":"load_dataset","repo":"adacheng/egothink","repo_kind":"listed","path":"gpt_eval.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/gpt_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27d8599384686c24"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"adacheng/egothink","repo_kind":"listed","path":"models/lego/mm_utils.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/lego/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"f04e8e006a8831c9","entry":"load_image_square","repo":"adacheng/egothink","repo_kind":"listed","path":"models/lego/mm_utils.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/lego/mm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f04e8e006a8831c9"}},{"code_sha256_prefix":"41fd9954784a1330","entry":"load_judge_prompts","repo":"adacheng/egothink","repo_kind":"listed","path":"common.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41fd9954784a1330"}},{"code_sha256_prefix":"270d3eb4371b0d85","entry":"load_model_answers","repo":"adacheng/egothink","repo_kind":"listed","path":"common.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"270d3eb4371b0d85"}},{"code_sha256_prefix":"88969cdc675c418e","entry":"load_questions","repo":"adacheng/egothink","repo_kind":"listed","path":"common.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88969cdc675c418e"}},{"code_sha256_prefix":"b8030ce2d52399ba","entry":"postprocess_box","repo":"adacheng/egothink","repo_kind":"listed","path":"models/lego/mm_utils.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/lego/mm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8030ce2d52399ba"}},{"code_sha256_prefix":"14a84c2cbfebc413","entry":"precompute_freqs_cis","repo":"adacheng/egothink","repo_kind":"listed","path":"models/llama_adapter_v2/llama.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/llama_adapter_v2/llama.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14a84c2cbfebc413"}},{"code_sha256_prefix":"70bf6ebaafd266c4","entry":"reshape_for_broadcast","repo":"adacheng/egothink","repo_kind":"listed","path":"models/llama_adapter_v2/llama.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/llama_adapter_v2/llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70bf6ebaafd266c4"}},{"code_sha256_prefix":"e252767324188623","entry":"round_by_factor","repo":"adacheng/egothink","repo_kind":"listed","path":"models/qwen_vl_utils_new.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/qwen_vl_utils_new.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e252767324188623"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"adacheng/egothink","repo_kind":"listed","path":"models/lego/utils.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/lego/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"adacheng/egothink","repo_kind":"listed","path":"models/lego/utils.py","file_url":"https://github.com/adacheng/egothink/blob/HEAD/models/lego/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}