{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clip-event-connecting-text-and-images-with","title":"CLIP-Event: Connecting Text and Images with Event Structures","arxiv_id":"2201.05078","date":"2022-01-13","proceeding":"CVPR 2022 1","authors":["Manling Li","Ruochen Xu","Shuohang Wang","Luowei Zhou","Xudong Lin","Chenguang Zhu","Michael Zeng","Heng Ji","Shih-Fu Chang"],"abstract":"Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding objects in images or entities in text, they often ignore the alignment at the level of events and their argument structures. In this work, we propose a contrastive learning framework to enforce vision-language pretraining models to comprehend events and associated argument (participant) roles. To achieve this, we take advantage of text information extraction technologies to obtain event structural knowledge, and utilize multiple prompt functions to contrast difficult negative descriptions by manipulating event structures. We also design an event graph alignment loss based on optimal transport to capture event argument structures. In addition, we collect a large event-rich dataset (106,875 images) for pretraining, which provides a more challenging image retrieval benchmark to assess the understanding of complicated lengthy sentences. Experiments show that our zero-shot CLIP-Event outperforms the state-of-the-art supervised model in argument extraction on Multimedia Event Extraction, achieving more than 5% absolute F-score gain in event extraction, as well as significant improvements on a variety of downstream tasks under zero-shot settings.","url_abs":"https://arxiv.org/abs/2201.05078v2","url_pdf":"https://arxiv.org/pdf/2201.05078v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clip-event-connecting-text-and-images-with","repo_url":"https://github.com/limanling/clip-event","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"clip-event-connecting-text-and-images-with","repo_url":"https://github.com/pwc-1/Paper-8/tree/main/clvp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"event-extraction","task_name":"Event Extraction"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2201.05078","atlas_url":"https://app.syntology.ai/?focus=2201.05078","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.05078"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/limanling/clip-event","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-8/tree/main/clvp","reach":null}],"summary":{"ran_fixture":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"9c9d2823696faa43","entry":"cost_matrix_cosine","repo":"limanling/clip-event","repo_kind":"official","path":"src/clip-event/model_ot.py","file_url":"https://github.com/limanling/clip-event/blob/HEAD/src/clip-event/model_ot.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c9d2823696faa43"}},{"code_sha256_prefix":"4c0a5849eef55a00","entry":"optimal_transport_dist","repo":"limanling/clip-event","repo_kind":"official","path":"src/clip-event/model_ot.py","file_url":"https://github.com/limanling/clip-event/blob/HEAD/src/clip-event/model_ot.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c0a5849eef55a00"}},{"code_sha256_prefix":"11728587f6b887ba","entry":"trace","repo":"limanling/clip-event","repo_kind":"official","path":"src/clip-event/model_ot.py","file_url":"https://github.com/limanling/clip-event/blob/HEAD/src/clip-event/model_ot.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11728587f6b887ba"}},{"code_sha256_prefix":"cbf43203110456f0","entry":"ipot","repo":"limanling/clip-event","repo_kind":"official","path":"src/clip-event/model_ot.py","file_url":"https://github.com/limanling/clip-event/blob/HEAD/src/clip-event/model_ot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbf43203110456f0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}