{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fine-tuning-clip-text-encoders-with-two-step","title":"Fine-tuning CLIP Text Encoders with Two-step Paraphrasing","arxiv_id":"2402.15120","date":"2024-02-23","proceeding":null,"authors":["Hyunjae Kim","Seunghyun Yoon","Trung Bui","Handong Zhao","Quan Tran","Franck Dernoncourt","Jaewoo Kang"],"abstract":"Contrastive language-image pre-training (CLIP) models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval, where the model is required to effectively process natural language input to produce an accurate visual output. However, current models still face limitations in dealing with linguistic variations in input queries, such as paraphrases, making it challenging to handle a broad range of user queries in real-world applications. In this study, we introduce a straightforward fine-tuning approach to enhance the representations of CLIP models for paraphrases. Our approach involves a two-step paraphrase generation process, where we automatically create two categories of paraphrases from web-scale image captions by leveraging large language models. Subsequently, we fine-tune the CLIP text encoder using these generated paraphrases while freezing the image encoder. Our resulting model, which we call ParaCLIP, exhibits significant improvements over baseline CLIP models across various tasks, including paraphrased retrieval (with rank similarity scores improved by up to 2.0% and 5.6%), Visual Genome Relation and Attribution, as well as seven semantic textual similarity tasks.","url_abs":"https://arxiv.org/abs/2402.15120v1","url_pdf":"https://arxiv.org/pdf/2402.15120v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"paraphrase-generation","task_name":"Paraphrase Generation"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.15120","atlas_url":"https://app.syntology.ai/?focus=2402.15120","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.15120"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mlfoundations/open_clip","reach":null}],"summary":{"ran":1,"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"found_in_text":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"bdfa567e4c5010da","entry":"GenLipLoss","repo":"mlfoundations/open_clip","repo_kind":"found_in_text","path":"src/open_clip/loss.py","file_url":"https://github.com/mlfoundations/open_clip/blob/HEAD/src/open_clip/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bdfa567e4c5010da"}},{"code_sha256_prefix":"efde606d2f18e7d1","entry":"caption_cross_entropy","repo":"mlfoundations/open_clip","repo_kind":"found_in_text","path":"src/open_clip/loss.py","file_url":"https://github.com/mlfoundations/open_clip/blob/HEAD/src/open_clip/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"efde606d2f18e7d1"}},{"code_sha256_prefix":"416baf9c9f70dfb7","entry":"resolve_caption_loss_dtype","repo":"mlfoundations/open_clip","repo_kind":"found_in_text","path":"src/open_clip/loss.py","file_url":"https://github.com/mlfoundations/open_clip/blob/HEAD/src/open_clip/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"416baf9c9f70dfb7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}