{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/segment-any-events-via-weighted-adaptation-of","title":"Segment Any Events via Weighted Adaptation of Pivotal Tokens","arxiv_id":"2312.16222","date":"2023-12-24","proceeding":null,"authors":["Zhiwen Chen","Zhiyu Zhu","Yifan Zhang","Junhui Hou","Guangming Shi","Jinjian Wu"],"abstract":"In this paper, we delve into the nuanced challenge of tailoring the Segment Anything Models (SAMs) for integration with event data, with the overarching objective of attaining robust and universal object segmentation within the event-centric domain. One pivotal issue at the heart of this endeavor is the precise alignment and calibration of embeddings derived from event-centric data such that they harmoniously coincide with those originating from RGB imagery. Capitalizing on the vast repositories of datasets with paired events and RGB images, our proposition is to harness and extrapolate the profound knowledge encapsulated within the pre-trained SAM framework. As a cornerstone to achieving this, we introduce a multi-scale feature distillation methodology. This methodology rigorously optimizes the alignment of token embeddings originating from event data with their RGB image counterparts, thereby preserving and enhancing the robustness of the overall architecture. Considering the distinct significance that token embeddings from intermediate layers hold for higher-level embeddings, our strategy is centered on accurately calibrating the pivotal token embeddings. This targeted calibration is aimed at effectively managing the discrepancies in high-level embeddings originating from both the event and image domains. Extensive experiments on different datasets demonstrate the effectiveness of the proposed distillation method. Code in http://github.com/happychenpipi/EventSAM.","url_abs":"https://arxiv.org/abs/2312.16222v1","url_pdf":"https://arxiv.org/pdf/2312.16222v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"segment-any-events-via-weighted-adaptation-of","repo_url":"https://github.com/happychenpipi/eventsam","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"event-based-object-segmentation","task_name":"Event-based Object Segmentation"}],"methods":[{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[{"slug":"ddd17-seg","name":"DDD17-SEG","full_name":""},{"slug":"dsec-seg","name":"DSEC-SEG","full_name":""},{"slug":"mvsec-seg","name":"MVSEC-SEG","full_name":""},{"slug":"rgbe-seg","name":"RGBE-SEG","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/event-based-object-segmentation-on-ddd17-seg","task":"Event-based Object Segmentation","dataset":"DDD17-SEG","model":"EventSAM","rank_in_archive_order":1,"of":2,"metrics":{"mIoU":"0.37"},"uses_additional_data":false},{"leaderboard":"/sota/event-based-object-segmentation-on-dsec-seg","task":"Event-based Object Segmentation","dataset":"DSEC-SEG","model":"EventSAM","rank_in_archive_order":1,"of":2,"metrics":{"mIoU":"0.38"},"uses_additional_data":false},{"leaderboard":"/sota/event-based-object-segmentation-on-mvsec-seg","task":"Event-based Object Segmentation","dataset":"MVSEC-SEG","model":"EventSAM","rank_in_archive_order":1,"of":8,"metrics":{"mIoU":"0.40"},"uses_additional_data":false},{"leaderboard":"/sota/event-based-object-segmentation-on-rgbe-seg","task":"Event-based Object Segmentation","dataset":"RGBE-SEG","model":"EventSAM","rank_in_archive_order":1,"of":8,"metrics":{"mIoU":"0.41"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.16222","atlas_url":"https://app.syntology.ai/?focus=2312.16222","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.16222"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/happychenpipi/eventsam","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_fixture":2,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d689131ee3dc4e0a","entry":"compute_attn_weighted_similarity_loss","repo":"happychenpipi/eventsam","repo_kind":"official","path":"event_encoder/loss/loss_function.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/event_encoder/loss/loss_function.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d689131ee3dc4e0a"}},{"code_sha256_prefix":"733d7f0bedcb74c2","entry":"get_rel_pos","repo":"happychenpipi/eventsam","repo_kind":"official","path":"segment_anything/modeling/image_encoder.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/segment_anything/modeling/image_encoder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"733d7f0bedcb74c2"}},{"code_sha256_prefix":"66e534e459b3b575","entry":"get_unique_colors","repo":"happychenpipi/eventsam","repo_kind":"official","path":"evaluate/calculate_metric.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/evaluate/calculate_metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66e534e459b3b575"}},{"code_sha256_prefix":"1629a39a857cb6d2","entry":"mask_from_color","repo":"happychenpipi/eventsam","repo_kind":"official","path":"evaluate/calculate_metric.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/evaluate/calculate_metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1629a39a857cb6d2"}},{"code_sha256_prefix":"a26e792dcda690cc","entry":"unique_colors_and_counts_in_part","repo":"happychenpipi/eventsam","repo_kind":"official","path":"evaluate/calculate_metric.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/evaluate/calculate_metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a26e792dcda690cc"}},{"code_sha256_prefix":"105fa08885dc36cc","entry":"window_partition","repo":"happychenpipi/eventsam","repo_kind":"official","path":"segment_anything/modeling/image_encoder.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/segment_anything/modeling/image_encoder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"105fa08885dc36cc"}},{"code_sha256_prefix":"f9bc5f31ce61cee8","entry":"build_sam_vit_b","repo":"happychenpipi/eventsam","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9bc5f31ce61cee8"}},{"code_sha256_prefix":"6b77c1f11fff3ed6","entry":"build_sam_vit_h","repo":"happychenpipi/eventsam","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b77c1f11fff3ed6"}},{"code_sha256_prefix":"3f8890e695469246","entry":"build_sam_vit_l","repo":"happychenpipi/eventsam","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f8890e695469246"}},{"code_sha256_prefix":"27be441cc8213e52","entry":"window_unpartition","repo":"happychenpipi/eventsam","repo_kind":"official","path":"segment_anything/modeling/image_encoder.py","file_url":"https://github.com/happychenpipi/eventsam/blob/HEAD/segment_anything/modeling/image_encoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27be441cc8213e52"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}