{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/centerclip-token-clustering-for-efficient","title":"CenterCLIP: Token Clustering for Efficient Text-Video Retrieval","arxiv_id":"2205.00823","date":"2022-05-02","proceeding":null,"authors":["Shuai Zhao","Linchao Zhu","Xiaohan Wang","Yi Yang"],"abstract":"Recently, large-scale pre-training methods like CLIP have made great progress in multi-modal research such as text-video retrieval. In CLIP, transformers are vital for modeling complex multi-modal relations. However, in the vision transformer of CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive and similar frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones. As the frame redundancy occurs mostly in consecutive frames, we divide videos into multiple segments and conduct segment-level clustering. Center tokens from each segment are later concatenated into a new sequence, while their original spatial-temporal relations are well maintained. We instantiate two clustering algorithms to efficiently find deterministic medoids and iteratively partition groups in high dimensional space. Through this token clustering and center selection procedure, we successfully reduce computation costs by removing redundant visual tokens. This method further enhances segment-level semantic alignment between video and text representations, enforcing the spatio-temporal interactions of tokens from within-segment frames. Our method, coined as CenterCLIP, surpasses existing state-of-the-art by a large margin on typical text-video benchmarks, while reducing the training memory cost by 35\\% and accelerating the inference speed by 14\\% at the best case. The code is available at \\href{{https://github.com/mzhaoshuai/CenterCLIP}}{{https://github.com/mzhaoshuai/CenterCLIP}}.","url_abs":"https://arxiv.org/abs/2205.00823v1","url_pdf":"https://arxiv.org/pdf/2205.00823v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"centerclip-token-clustering-for-efficient","repo_url":"https://github.com/mzhaoshuai/CenterCLIP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"CenterCLIP (ViT-B/16)","rank_in_archive_order":20,"of":31,"metrics":{"text-to-video Mean Rank":"5.7","text-to-video Median Rank":"2","text-to-video R@1":"46.2","text-to-video R@10":"87.6","text-to-video R@5":"77.0","video-to-text Mean Rank":"5.5","video-to-text Median Rank":"2","video-to-text R@1":"46.7","video-to-text R@10":"88.0","video-to-text R@5":"77.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"CenterCLIP (ViT-B/16)","rank_in_archive_order":20,"of":38,"metrics":{"text-to-video Mean Rank":"47.3","text-to-video Median Rank":"8","text-to-video R@1":"24.2","text-to-video R@10":"55.9","text-to-video R@5":"46.2","video-to-text Mean Rank":"41.3","video-to-text Median Rank":"7","video-to-text R@1":"24.5","video-to-text R@10":"55.8","video-to-text R@5":"46.4"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"CenterCLIP (ViT-B/16)","rank_in_archive_order":26,"of":63,"metrics":{"text-to-video Mean Rank":"13.8","text-to-video Median Rank":"2","text-to-video R@1":"48.4","text-to-video R@10":"82.0","text-to-video R@5":"73.8","video-to-text Mean Rank":"10.2","video-to-text Median Rank":"2","video-to-text R@1":"47.7","video-to-text R@10":"83.3","video-to-text R@5":"75.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msvd","task":"Video Retrieval","dataset":"MSVD","model":"CenterCLIP (ViT-B/16)","rank_in_archive_order":11,"of":24,"metrics":{"text-to-video Mean Rank":"8.4","text-to-video Median Rank":"1","text-to-video R@1":"50.6","text-to-video R@10":"88.4","text-to-video R@5":"80.3","video-to-text Mean Rank":"3.0","video-to-text Median Rank":"1","video-to-text R@1":"68.4","video-to-text R@10":"95.0","video-to-text R@5":"90.1"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.00823","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.00823"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mzhaoshuai/CenterCLIP","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"93f1c0cd5419e51b","entry":"get_cluster_inter","repo":"mzhaoshuai/CenterCLIP","repo_kind":"official","path":"modules/cluster/cluster.py","file_url":"https://github.com/mzhaoshuai/CenterCLIP/blob/HEAD/modules/cluster/cluster.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"93f1c0cd5419e51b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}