{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vision-language-pre-training-with-triple","title":"Vision-Language Pre-Training with Triple Contrastive Learning","arxiv_id":"2202.10401","date":"2022-02-21","proceeding":"CVPR 2022 1","authors":["Jinyu Yang","Jiali Duan","Son Tran","Yi Xu","Sampath Chanda","Liqun Chen","Belinda Zeng","Trishul Chilimbi","Junzhou Huang"],"abstract":"Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss). The success of this alignment strategy is attributed to its capability in maximizing the mutual information (MI) between an image and its matched text. However, simply performing cross-modal alignment (CMA) ignores data potential within each modality, which may result in degraded representations. For instance, although CMA-based models are able to map image-text pairs close together in the embedding space, they fail to ensure that similar inputs from the same modality stay close by. This problem can get even worse when the pre-training data is noisy. In this paper, we propose triple contrastive learning (TCL) for vision-language pre-training by leveraging both cross-modal and intra-modal self-supervision. Besides CMA, TCL introduces an intra-modal contrastive objective to provide complementary benefits in representation learning. To take advantage of localized and structural information from image and text input, TCL further maximizes the average MI between local regions of image/text and their global summary. To the best of our knowledge, ours is the first work that takes into account local structure information for multi-modality representation learning. Experimental evaluations show that our approach is competitive and achieves the new state of the art on various common down-stream vision-language tasks such as image-text retrieval and visual question answering.","url_abs":"https://arxiv.org/abs/2202.10401v4","url_pdf":"https://arxiv.org/pdf/2202.10401v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vision-language-pre-training-with-triple","repo_url":"https://github.com/uta-smile/TCL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-cross-modal-retrieval","task_name":"Zero-Shot Cross-Modal Retrieval"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"infonce","method_name":"InfoNCE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"TCL","rank_in_archive_order":18,"of":36,"metrics":{"Image-to-text R@1":"75.6","Image-to-text R@10":"96.7","Image-to-text R@5":"92.8","Text-to-image R@1":"59.0","Text-to-image R@10":"89.9","Text-to-image R@5":"83.2"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-cross-modal-retrieval-on-coco-2014","task":"Zero-Shot Cross-Modal Retrieval","dataset":"COCO 2014","model":"TCL","rank_in_archive_order":3,"of":18,"metrics":{"Image-to-text R@1":"71.4","Image-to-text R@10":"95.4","Image-to-text R@5":"90.8","Text-to-image R@1":"53.5","Text-to-image R@10":"87.1","Text-to-image R@5":"79.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.10401","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.10401"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/uta-smile/TCL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c7f85279d772ea19","entry":"itm_eval","repo":"uta-smile/TCL","repo_kind":"official","path":"Retrieval.py","file_url":"https://github.com/uta-smile/TCL/blob/HEAD/Retrieval.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7f85279d772ea19"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"uta-smile/TCL","repo_kind":"official","path":"models/model_pretrain.py","file_url":"https://github.com/uta-smile/TCL/blob/HEAD/models/model_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}