{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/expectation-maximization-contrastive-learning","title":"Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations","arxiv_id":"2211.11427","date":"2022-11-21","proceeding":null,"authors":["Peng Jin","Jinfa Huang","Fenglin Liu","Xian Wu","Shen Ge","Guoli Song","David A. Clifton","Jie Chen"],"abstract":"Most video-and-language representation learning approaches employ contrastive learning, e.g., CLIP, to project the video and text features into a common latent space according to the semantic similarities of text-video pairs. However, such learned shared latent spaces are not often optimal, and the modality gap between visual and textual representation can not be fully eliminated. In this paper, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations. Specifically, we use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. Such feature decomposition of video-and-language representations reduces the rank of the latent space, resulting in increased representing power for the semantics. Extensive experiments on three benchmark text-video retrieval datasets prove that our EMCL can learn more discriminative video-and-language representations than previous methods, and significantly outperform previous state-of-the-art methods across all metrics. More encouragingly, the proposed method can be applied to boost the performance of existing approaches either as a jointly training layer or an out-of-the-box inference module with no extra training, making it easy to be incorporated into any existing methods.","url_abs":"https://arxiv.org/abs/2211.11427v1","url_pdf":"https://arxiv.org/pdf/2211.11427v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"expectation-maximization-contrastive-learning","repo_url":"https://github.com/jpthu17/emcl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"expectation-maximization-contrastive-learning","repo_url":"https://github.com/jpthu17/HBI","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"expectation-maximization-contrastive-learning","repo_url":"https://github.com/jpthu17/dicosa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"expectation-maximization-contrastive-learning","repo_url":"https://github.com/jpthu17/diffusionret","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-captioning-on-msr-vtt-1","task":"Video Captioning","dataset":"MSR-VTT","model":"EMCL-Net","rank_in_archive_order":22,"of":24,"metrics":{"BLEU-4":"45.3","CIDEr":"54.6","METEOR":"30.2","ROUGE-L":"63.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-msrvtt-qa","task":"Video Question Answering","dataset":"MSRVTT-QA","model":"EMCL-Net","rank_in_archive_order":9,"of":14,"metrics":{"Accuracy":"45.8"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"EMCL-Net++","rank_in_archive_order":16,"of":31,"metrics":{"text-to-video Mean Rank":"1","text-to-video R@1":"50.6","text-to-video R@5":"78.7","text-to-video R@50":"98.1","video-to-text Mean Rank":"1","video-to-text R@1":"50.6","video-to-text R@5":"78.9","video-to-text R@50":"98.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"EMCL-Net","rank_in_archive_order":24,"of":31,"metrics":{"text-to-video Mean Rank":"2","text-to-video R@1":"41.2","text-to-video R@5":"72.7","video-to-text Mean Rank":"2","video-to-text R@1":"42.7","video-to-text R@5":"74","video-to-text R@50":"98.3"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"EMCL-Net++","rank_in_archive_order":15,"of":38,"metrics":{"text-to-video R@1":"25.9","text-to-video R@5":"46.4","video-to-text Mean Rank":"8","video-to-text R@1":"26.7","video-to-text R@10":"54.4","video-to-text R@5":"44.7"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"EMCL-Net","rank_in_archive_order":22,"of":38,"metrics":{"text-to-video R@1":"23.9","text-to-video R@10":"50.9","text-to-video R@5":"42.4","video-to-text Mean Rank":"12","video-to-text R@1":"22.2","video-to-text R@10":"49.2","video-to-text R@5":"40.6"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"EMCL-Net (Ours)++ LSMDC Rohrbach et al. (2015)","rank_in_archive_order":38,"of":38,"metrics":{"text-to-video Mean Rank":"8","text-to-video R@10":"53.7"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"EMCL-Net++","rank_in_archive_order":15,"of":63,"metrics":{"text-to-video Mean Rank":"1","text-to-video R@1":"51.6","text-to-video R@10":"85.3","text-to-video R@5":"78.1","video-to-text Mean Rank":"1","video-to-text R@1":"51.8","video-to-text R@10":"88","video-to-text R@5":"80.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"EMCL-Net","rank_in_archive_order":32,"of":63,"metrics":{"text-to-video Mean Rank":"2","text-to-video R@1":"46.8","text-to-video R@10":"83.1","text-to-video R@5":"73.1","video-to-text Mean Rank":"2","video-to-text R@1":"46.5","video-to-text R@10":"83.5","video-to-text R@5":"73.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"EMCL-Net","rank_in_archive_order":13,"of":34,"metrics":{"Accuracy":"0.458"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.11427","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.11427"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/dicosa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/HBI","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/emcl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/diffusionret","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f09b191d136f2213","entry":"cluster_dpc_knn","repo":"jpthu17/HBI","repo_kind":"listed","path":"HBI/models/cluster.py","file_url":"https://github.com/jpthu17/HBI/blob/HEAD/HBI/models/cluster.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f09b191d136f2213"}},{"code_sha256_prefix":"eba8c64dc2ab559d","entry":"get_args","repo":"jpthu17/EMCL","repo_kind":"official","path":"video_retrieval/EMCL-Net/main_retrieval.py","file_url":"https://github.com/jpthu17/EMCL/blob/HEAD/video_retrieval/EMCL-Net/main_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eba8c64dc2ab559d"}},{"code_sha256_prefix":"a1d91d4a1b5bbd6f","entry":"index_points","repo":"jpthu17/HBI","repo_kind":"listed","path":"HBI/models/cluster.py","file_url":"https://github.com/jpthu17/HBI/blob/HEAD/HBI/models/cluster.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1d91d4a1b5bbd6f"}},{"code_sha256_prefix":"8952edef143f2747","entry":"set_seed_logger","repo":"jpthu17/EMCL","repo_kind":"official","path":"video_retrieval/EMCL-Net/main_retrieval.py","file_url":"https://github.com/jpthu17/EMCL/blob/HEAD/video_retrieval/EMCL-Net/main_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8952edef143f2747"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}