{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-11417","title":"Robust Ego-Exo Correspondence with Long-Term Memory","arxiv_id":"2510.11417","date":"2025-10-13","proceeding":"NeurIPS","authors":["Yijun Hu","Bing Fan","Xin Gu","Haiqing Ren","Dongfang Liu","Heng Fan","Libo Zhang"],"abstract":"Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint variations, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM 2 encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a dual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Experts (MoE). Compared to SAM 2, our approach features (i) a Memory-View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and (ii) a dual-memory bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed LM-EEC, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at https://github.com/juneyeeHu/LM-EEC.","url_abs":"https://arxiv.org/abs/2510.11417","url_pdf":"https://arxiv.org/pdf/2510.11417","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2510.11417","atlas_url":"https://app.syntology.ai/?focus=2510.11417","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.11417"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/juneyeeHu/LM-EEC","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran_honours":1,"ran_fixture":1,"unverified":11},"by_repo_kind":{"found_in_text":{"samples":16,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"7ceaad42da42edd5","entry":"compute_axial_cis","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ceaad42da42edd5"}},{"code_sha256_prefix":"a6c3974188a5ffb0","entry":"get_1d_sine_pe","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6c3974188a5ffb0"}},{"code_sha256_prefix":"bb060d29525f90eb","entry":"init_t_xy","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb060d29525f90eb"}},{"code_sha256_prefix":"f8f09f78b474ee7c","entry":"reshape_for_broadcast","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8f09f78b474ee7c"}},{"code_sha256_prefix":"4ee20f090e58c425","entry":"select_closest_cond_frames","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ee20f090e58c425"}},{"code_sha256_prefix":"1f9402f6f57d17a3","entry":"build_sam2","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/build_sam.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f9402f6f57d17a3"}},{"code_sha256_prefix":"0bf83f3242d73acc","entry":"build_sam2_video_predictor","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/build_sam.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bf83f3242d73acc"}},{"code_sha256_prefix":"da8bc97ac60e4aa7","entry":"build_sam2_video_predictor_ego","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/build_sam.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da8bc97ac60e4aa7"}},{"code_sha256_prefix":"e473e66fedc883c7","entry":"dice_loss","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"training/loss_fns.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/training/loss_fns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e473e66fedc883c7"}},{"code_sha256_prefix":"f2e72d81f39d65cd","entry":"get_activation_fn","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2e72d81f39d65cd"}},{"code_sha256_prefix":"36d321604b37dd80","entry":"get_folders","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"evaluation/merge_pred.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/evaluation/merge_pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36d321604b37dd80"}},{"code_sha256_prefix":"b8f501ca29e77f1a","entry":"iou_loss","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"training/loss_fns.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/training/loss_fns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8f501ca29e77f1a"}},{"code_sha256_prefix":"74b02d33b0122c24","entry":"kmeans_cluster_frame","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/sam2_base.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/sam2_base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74b02d33b0122c24"}},{"code_sha256_prefix":"4cd943feb67480e9","entry":"memory_bank_compress","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"sam2/modeling/sam2_base.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/sam2/modeling/sam2_base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cd943feb67480e9"}},{"code_sha256_prefix":"523182971553fa2c","entry":"process_take","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"evaluation/merge_pred.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/evaluation/merge_pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"523182971553fa2c"}},{"code_sha256_prefix":"42fec6fdc1faf6f1","entry":"sigmoid_focal_loss","repo":"juneyeeHu/LM-EEC","repo_kind":"found_in_text","path":"training/loss_fns.py","file_url":"https://github.com/juneyeeHu/LM-EEC/blob/HEAD/training/loss_fns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42fec6fdc1faf6f1"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}