{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-clip-training-with-language-1","title":"Improving CLIP Training with Language Rewrites","arxiv_id":"2305.20088","date":"2023-05-31","proceeding":"NeurIPS 2023 11","authors":["Lijie Fan","Dilip Krishnan","Phillip Isola","Dina Katabi","Yonglong Tian"],"abstract":"Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.","url_abs":"https://arxiv.org/abs/2305.20088v2","url_pdf":"https://arxiv.org/pdf/2305.20088v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-clip-training-with-language-1","repo_url":"https://github.com/lijiefan/laclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.20088","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.20088"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LijieFan/LaCLIP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lijiefan/laclip","reach":null}],"summary":{"ran":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b781cd857f7e6956","entry":"CsvDatasetAugCap","repo":"lijiefan/laclip","repo_kind":"official","path":"data.py","file_url":"https://github.com/lijiefan/laclip/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b781cd857f7e6956"}},{"code_sha256_prefix":"c2c68e245f932a2c","entry":"validate_zeroshot","repo":"LijieFan/LaCLIP","repo_kind":"official","path":"eval_zeroshot_imagenet.py","file_url":"https://github.com/LijieFan/LaCLIP/blob/HEAD/eval_zeroshot_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c2c68e245f932a2c"}},{"code_sha256_prefix":"626a720ea2f648be","entry":"get_model","repo":"LijieFan/LaCLIP","repo_kind":"official","path":"eval_zeroshot_imagenet.py","file_url":"https://github.com/LijieFan/LaCLIP/blob/HEAD/eval_zeroshot_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"626a720ea2f648be"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}