{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tempme-video-temporal-token-merging-for","title":"TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval","arxiv_id":"2409.01156","date":"2024-09-02","proceeding":null,"authors":["Leqi Shen","Tianxiang Hao","Sicheng Zhao","Yifeng Zhang","Pengzhang Liu","Yongjun Bao","Guiguang Ding"],"abstract":"Most text-video retrieval methods utilize the text-image pre-trained CLIP as a backbone, incorporating complex modules that result in high computational overhead. As a result, many studies focus on efficient fine-tuning. The primary challenge in efficient adaption arises from the inherent differences between image and video modalities. Each sampled video frame must be processed by the image encoder independently, which increases complexity and complicates practical deployment. Although existing efficient methods fine-tune with small trainable parameters, they still incur high inference costs due to the large token number. In this work, we argue that temporal redundancy significantly contributes to the model's high complexity due to the repeated information in consecutive frames. Existing token compression methods for image models fail to solve the unique challenges, as they overlook temporal redundancy across frames. To tackle these problems, we propose Temporal Token Merging (TempMe) to reduce temporal redundancy. Specifically, we introduce a progressive multi-granularity framework. By gradually combining neighboring clips, we merge temporal tokens across different frames and learn video-level features, leading to lower complexity and better performance. Extensive experiments validate the superiority of our TempMe. Compared to previous efficient text-video retrieval methods, TempMe significantly reduces output tokens by 95% and GFLOPs by 51%, while achieving a 1.8X speedup and a 4.4% R-Sum improvement. Additionally, TempMe exhibits robust generalization capabilities by integrating effectively with both efficient and full fine-tuning methods. With full fine-tuning, TempMe achieves a significant 7.9% R-Sum improvement, trains 1.57X faster, and utilizes 75.2% GPU memory usage. Our code will be released.","url_abs":"https://arxiv.org/abs/2409.01156v1","url_pdf":"https://arxiv.org/pdf/2409.01156v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tempme-video-temporal-token-merging-for","repo_url":"https://github.com/lunarshen/dsicovla","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.01156","atlas_url":"https://app.syntology.ai/?focus=2409.01156","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.01156"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LunarShen/TempMe","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lunarshen/dsicovla","reach":null}],"summary":{"ran":3,"ran_fixture":1,"ran_honours":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1},"found_in_text":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cbfdf39eacd6be6c","entry":"Attention","repo":"LunarShen/TempMe","repo_kind":"found_in_text","path":"tvr/models/module_tome_patch.py","file_url":"https://github.com/LunarShen/TempMe/blob/HEAD/tvr/models/module_tome_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbfdf39eacd6be6c"}},{"code_sha256_prefix":"f770b619ad6d77b5","entry":"Attention_GlobalCLS","repo":"lunarshen/dsicovla","repo_kind":"listed","path":"tvr/models/module_transformer.py","file_url":"https://github.com/lunarshen/dsicovla/blob/HEAD/tvr/models/module_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f770b619ad6d77b5"}},{"code_sha256_prefix":"07221ceabbfc85ec","entry":"ResidualAttentionBlock","repo":"LunarShen/TempMe","repo_kind":"found_in_text","path":"tvr/models/module_tome_patch.py","file_url":"https://github.com/LunarShen/TempMe/blob/HEAD/tvr/models/module_tome_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07221ceabbfc85ec"}},{"code_sha256_prefix":"c8bfd649bf334138","entry":"bipartite_soft_matching","repo":"LunarShen/TempMe","repo_kind":"found_in_text","path":"tvr/models/module_tome_patch.py","file_url":"https://github.com/LunarShen/TempMe/blob/HEAD/tvr/models/module_tome_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8bfd649bf334138"}},{"code_sha256_prefix":"969e8250d805a154","entry":"do_nothing","repo":"LunarShen/TempMe","repo_kind":"found_in_text","path":"tvr/models/module_tome_patch.py","file_url":"https://github.com/LunarShen/TempMe/blob/HEAD/tvr/models/module_tome_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"969e8250d805a154"}},{"code_sha256_prefix":"d6d31b246591a6d0","entry":"merge_source","repo":"LunarShen/TempMe","repo_kind":"found_in_text","path":"tvr/models/module_tome_patch.py","file_url":"https://github.com/LunarShen/TempMe/blob/HEAD/tvr/models/module_tome_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6d31b246591a6d0"}},{"code_sha256_prefix":"726064eec41365b4","entry":"merge_wavg","repo":"LunarShen/TempMe","repo_kind":"found_in_text","path":"tvr/models/module_tome_patch.py","file_url":"https://github.com/LunarShen/TempMe/blob/HEAD/tvr/models/module_tome_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"726064eec41365b4"}},{"code_sha256_prefix":"f46324f012ea5ba4","entry":"ToMeBlock","repo":"LunarShen/TempMe","repo_kind":"found_in_text","path":"tvr/models/module_tome_patch.py","file_url":"https://github.com/LunarShen/TempMe/blob/HEAD/tvr/models/module_tome_patch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f46324f012ea5ba4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}