{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/long-clip-unlocking-the-long-text-capability","title":"Long-CLIP: Unlocking the Long-Text Capability of CLIP","arxiv_id":"2403.15378","date":"2024-03-22","proceeding":null,"authors":["Beichen Zhang","Pan Zhang","Xiaoyi Dong","Yuhang Zang","Jiaqi Wang"],"abstract":"Contrastive Language-Image Pre-training (CLIP) has been the cornerstone for zero-shot classification, text-image retrieval, and text-image generation by aligning image and text modalities. Despite its widespread adoption, a significant limitation of CLIP lies in the inadequate length of text input. The length of the text token is restricted to 77, and an empirical study shows the actual effective length is even less than 20. This prevents CLIP from handling detailed descriptions, limiting its applications for image retrieval and text-to-image generation with extensive prerequisites. To this end, we propose Long-CLIP as a plug-and-play alternative to CLIP that supports long-text input, retains or even surpasses its zero-shot generalizability, and aligns the CLIP latent space, making it readily replace CLIP without any further adaptation in downstream frameworks. Nevertheless, achieving this goal is far from straightforward, as simplistic fine-tuning can result in a significant degradation of CLIP's performance. Moreover, substituting the text encoder with a language model supporting longer contexts necessitates pretraining with vast amounts of data, incurring significant expenses. Accordingly, Long-CLIP introduces an efficient fine-tuning solution on CLIP with two novel strategies designed to maintain the original capabilities, including (1) a knowledge-preserved stretching of positional embedding and (2) a primary component matching of CLIP features. With leveraging just one million extra long text-image pairs, Long-CLIP has shown the superiority to CLIP for about 20% in long caption text-image retrieval and 6% in traditional text-image retrieval tasks, e.g., COCO and Flickr30k. Furthermore, Long-CLIP offers enhanced capabilities for generating images from detailed text descriptions by replacing CLIP in a plug-and-play manner.","url_abs":"https://arxiv.org/abs/2403.15378v3","url_pdf":"https://arxiv.org/pdf/2403.15378v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"long-clip-unlocking-the-long-text-capability","repo_url":"https://github.com/beichenzbc/long-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.15378","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.15378"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/beichenzbc/long-clip","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":2,"ran":1,"unverified":10},"by_repo_kind":{"official":{"samples":15,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f19962ebb134b3d7","entry":"convert_to_custom_text_state_dict","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/model.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19962ebb134b3d7"}},{"code_sha256_prefix":"4bdd36eab04c3f62","entry":"get_1d_sincos_pos_embed_from_grid","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/pos_embed.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/pos_embed.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4bdd36eab04c3f62"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/model.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"b476c8cfbf0f1f47","entry":"get_input_dtype","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/model.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b476c8cfbf0f1f47"}},{"code_sha256_prefix":"0b94bc3f6eb369a4","entry":"load_state_dict","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/factory.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b94bc3f6eb369a4"}},{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/loss.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"9417ae492629cf06","entry":"get_2d_sincos_pos_embed","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/pos_embed.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9417ae492629cf06"}},{"code_sha256_prefix":"4fd80de79832745d","entry":"get_2d_sincos_pos_embed_from_grid","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/pos_embed.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/pos_embed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4fd80de79832745d"}},{"code_sha256_prefix":"5c7cf820e99443ff","entry":"get_model_config","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/factory.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c7cf820e99443ff"}},{"code_sha256_prefix":"2665f49f3b92c031","entry":"get_tokenizer","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/factory.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2665f49f3b92c031"}},{"code_sha256_prefix":"8811c5d06f3a2eaa","entry":"load_openai_model","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/openai.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/openai.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8811c5d06f3a2eaa"}},{"code_sha256_prefix":"e332856e3c2fc814","entry":"neighbour_exchange","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/loss.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e332856e3c2fc814"}},{"code_sha256_prefix":"5b1fd364afcf3c05","entry":"neighbour_exchange_bidir","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/loss.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b1fd364afcf3c05"}},{"code_sha256_prefix":"fb651d0a97fd4d3f","entry":"prepare_inputs_for_generation","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/coca_model.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/coca_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb651d0a97fd4d3f"}},{"code_sha256_prefix":"2a377da4a76a2d44","entry":"register_pooler","repo":"beichenzbc/long-clip","repo_kind":"official","path":"open_clip_long/hf_model.py","file_url":"https://github.com/beichenzbc/long-clip/blob/HEAD/open_clip_long/hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a377da4a76a2d44"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}