{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efm3d-a-benchmark-for-measuring-progress","title":"EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Models","arxiv_id":"2406.10224","date":"2024-06-14","proceeding":null,"authors":["Julian Straub","Daniel DeTone","Tianwei Shen","Nan Yang","Chris Sweeney","Richard Newcombe"],"abstract":"The advent of wearable computers enables a new source of context for AI that is embedded in egocentric sensor data. This new egocentric data comes equipped with fine-grained 3D location information and thus presents the opportunity for a novel class of spatial foundation models that are rooted in 3D space. To measure progress on what we term Egocentric Foundation Models (EFMs) we establish EFM3D, a benchmark with two core 3D egocentric perception tasks. EFM3D is the first benchmark for 3D object detection and surface regression on high quality annotated egocentric data of Project Aria. We propose Egocentric Voxel Lifting (EVL), a baseline for 3D EFMs. EVL leverages all available egocentric modalities and inherits foundational capabilities from 2D foundation models. This model, trained on a large simulated dataset, outperforms existing methods on the EFM3D benchmark.","url_abs":"https://arxiv.org/abs/2406.10224v1","url_pdf":"https://arxiv.org/pdf/2406.10224v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efm3d-a-benchmark-for-measuring-progress","repo_url":"https://github.com/facebookresearch/efm3d","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"3d-reconstruction","task_name":"3D Reconstruction"},{"task_slug":"multi-view-3d-reconstruction","task_name":"Multi-View 3D Reconstruction"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[{"slug":"aria-everyday-objects","name":"Aria Everyday Objects","full_name":"Aria Everyday Objects"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-object-detection-on-aeo","task":"3D Object Detection","dataset":"Aria Everyday Objects","model":"EVL","rank_in_archive_order":1,"of":4,"metrics":{"mAP":"22"},"uses_additional_data":false},{"leaderboard":"/sota/3d-object-detection-on-aeo","task":"3D Object Detection","dataset":"Aria Everyday Objects","model":"3DETR","rank_in_archive_order":2,"of":4,"metrics":{"mAP":"16"},"uses_additional_data":false},{"leaderboard":"/sota/3d-object-detection-on-aeo","task":"3D Object Detection","dataset":"Aria Everyday Objects","model":"ImVoxelNet","rank_in_archive_order":3,"of":4,"metrics":{"mAP":"15"},"uses_additional_data":false},{"leaderboard":"/sota/3d-object-detection-on-aeo","task":"3D Object Detection","dataset":"Aria Everyday Objects","model":"Cube R-CNN","rank_in_archive_order":4,"of":4,"metrics":{"mAP":"8"},"uses_additional_data":false},{"leaderboard":"/sota/3d-object-detection-on-aria-synthetic","task":"3D Object Detection","dataset":"Aria Synthetic Environments","model":"EVL","rank_in_archive_order":1,"of":4,"metrics":{"MAP":"75"},"uses_additional_data":false},{"leaderboard":"/sota/3d-object-detection-on-aria-synthetic","task":"3D Object Detection","dataset":"Aria Synthetic Environments","model":"ImVoxelNet","rank_in_archive_order":2,"of":4,"metrics":{"MAP":"64"},"uses_additional_data":false},{"leaderboard":"/sota/3d-object-detection-on-aria-synthetic","task":"3D Object Detection","dataset":"Aria Synthetic Environments","model":"Cube R-CNN","rank_in_archive_order":3,"of":4,"metrics":{"MAP":"36"},"uses_additional_data":false},{"leaderboard":"/sota/3d-object-detection-on-aria-synthetic","task":"3D Object Detection","dataset":"Aria Synthetic Environments","model":"3DETR","rank_in_archive_order":4,"of":4,"metrics":{"MAP":"33"},"uses_additional_data":false},{"leaderboard":"/sota/3d-reconstruction-on-aria-digital-twin","task":"3D Reconstruction","dataset":"Aria Digital Twin Dataset","model":"EVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"18.2","Completeness":"3.105","Precision":"59.4"},"uses_additional_data":false},{"leaderboard":"/sota/3d-reconstruction-on-aria-synthetic","task":"3D Reconstruction","dataset":"Aria Synthetic Environments","model":"EVL","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"5.7","Completeness":"87.7","Precision":"82.2","Recall":"10.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.10224","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.10224"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/efm3d","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3f66b5f677e07adc","entry":"add_residual","repo":"facebookresearch/efm3d","repo_kind":"official","path":"efm3d/model/dinov2_utils.py","file_url":"https://github.com/facebookresearch/efm3d/blob/HEAD/efm3d/model/dinov2_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f66b5f677e07adc"}},{"code_sha256_prefix":"4d7d35dca4659acd","entry":"drop_add_residual_stochastic_depth","repo":"facebookresearch/efm3d","repo_kind":"official","path":"efm3d/model/dinov2_utils.py","file_url":"https://github.com/facebookresearch/efm3d/blob/HEAD/efm3d/model/dinov2_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d7d35dca4659acd"}},{"code_sha256_prefix":"de16c8c98b583b6a","entry":"get_branges_scales","repo":"facebookresearch/efm3d","repo_kind":"official","path":"efm3d/model/dinov2_utils.py","file_url":"https://github.com/facebookresearch/efm3d/blob/HEAD/efm3d/model/dinov2_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de16c8c98b583b6a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}