{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cm3ae-a-unified-rgb-frame-and-event-voxel","title":"CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework","arxiv_id":"2504.12576","date":"2025-04-17","proceeding":null,"authors":["Wentao Wu","Xiao Wang","Chenglong Li","Bo Jiang","Jin Tang","Bin Luo","Qi Liu"],"abstract":"Event cameras have attracted increasing attention in recent years due to their advantages in high dynamic range, high temporal resolution, low power consumption, and low latency. Some researchers have begun exploring pre-training directly on event data. Nevertheless, these efforts often fail to establish strong connections with RGB frames, limiting their applicability in multi-modal fusion scenarios. To address these issues, we propose a novel CM3AE pre-training framework for the RGB-Event perception. This framework accepts multi-modalities/views of data as input, including RGB images, event images, and event voxels, providing robust support for both event-based and RGB-event fusion based downstream tasks. Specifically, we design a multi-modal fusion reconstruction module that reconstructs the original image from fused multi-modal features, explicitly enhancing the model's ability to aggregate cross-modal complementary information. Additionally, we employ a multi-modal contrastive learning strategy to align cross-modal feature representations in a shared latent space, which effectively enhances the model's capability for multi-modal understanding and capturing global dependencies. We construct a large-scale dataset containing 2,535,759 RGB-Event data pairs for the pre-training. Extensive experiments on five downstream tasks fully demonstrated the effectiveness of CM3AE. Source code and pre-trained models will be released on https://github.com/Event-AHU/CM3AE.","url_abs":"https://arxiv.org/abs/2504.12576v1","url_pdf":"https://arxiv.org/pdf/2504.12576v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cm3ae-a-unified-rgb-frame-and-event-voxel","repo_url":"https://github.com/event-ahu/cm3ae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.12576","atlas_url":"https://app.syntology.ai/?focus=2504.12576","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.12576"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/event-ahu/cm3ae","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_honours":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3dc19396537db789","entry":"all_reduce_mean","repo":"event-ahu/cm3ae","repo_kind":"official","path":"misc.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dc19396537db789"}},{"code_sha256_prefix":"4d25be8ebf17fde1","entry":"bool_flag","repo":"event-ahu/cm3ae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d25be8ebf17fde1"}},{"code_sha256_prefix":"2b5489053b222e00","entry":"clip_gradients","repo":"event-ahu/cm3ae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b5489053b222e00"}},{"code_sha256_prefix":"361a6b24f11fc50a","entry":"cosine_scheduler","repo":"event-ahu/cm3ae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"361a6b24f11fc50a"}},{"code_sha256_prefix":"c2c3b52c5e6528a5","entry":"csv_Dataset","repo":"event-ahu/cm3ae","repo_kind":"official","path":"OurDataset.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/OurDataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2c3b52c5e6528a5"}},{"code_sha256_prefix":"67e6de7410ae3cde","entry":"find_stage_index","repo":"event-ahu/cm3ae","repo_kind":"official","path":"misc.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67e6de7410ae3cde"}},{"code_sha256_prefix":"4bdd36eab04c3f62","entry":"get_1d_sincos_pos_embed_from_grid","repo":"event-ahu/cm3ae","repo_kind":"official","path":"pos_embed.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/pos_embed.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bdd36eab04c3f62"}},{"code_sha256_prefix":"a5d41302c947503d","entry":"get_params","repo":"event-ahu/cm3ae","repo_kind":"official","path":"transform.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/transform.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5d41302c947503d"}},{"code_sha256_prefix":"1ae0a7614fdc48ab","entry":"resize_bbox","repo":"event-ahu/cm3ae","repo_kind":"official","path":"transform.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/transform.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ae0a7614fdc48ab"}},{"code_sha256_prefix":"9417ae492629cf06","entry":"get_2d_sincos_pos_embed","repo":"event-ahu/cm3ae","repo_kind":"official","path":"pos_embed.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9417ae492629cf06"}},{"code_sha256_prefix":"4fd80de79832745d","entry":"get_2d_sincos_pos_embed_from_grid","repo":"event-ahu/cm3ae","repo_kind":"official","path":"pos_embed.py","file_url":"https://github.com/event-ahu/cm3ae/blob/HEAD/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fd80de79832745d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}