{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tulip-token-length-upgraded-clip","title":"TULIP: Token-length Upgraded CLIP","arxiv_id":"2410.10034","date":"2024-10-13","proceeding":null,"authors":["Ivona Najdenkoska","Mohammad Mahdi Derakhshani","Yuki M. Asano","Nanne van Noord","Marcel Worring","Cees G. M. Snoek"],"abstract":"We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip.","url_abs":"https://arxiv.org/abs/2410.10034v2","url_pdf":"https://arxiv.org/pdf/2410.10034v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tulip-token-length-upgraded-clip","repo_url":"https://github.com/ivonajdenkoska/tulip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":null,"task_name":"Position"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"relative-position-encodings","method_name":"Relative Position Encodings"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.10034","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.10034"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ivonajdenkoska/tulip","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":4,"ran":6,"ran_violates":1,"unverified":9},"by_repo_kind":{"official":{"samples":20,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a677092c48744688","entry":"natural_key","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval_tulip.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval_tulip.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a677092c48744688"}},{"code_sha256_prefix":"09c78a125c9f58ae","entry":"collate_fn","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/dci_long.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/dci_long.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09c78a125c9f58ae"}},{"code_sha256_prefix":"f19962ebb134b3d7","entry":"convert_to_custom_text_state_dict","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/model.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19962ebb134b3d7"}},{"code_sha256_prefix":"385a40328778e8cb","entry":"get_accuracy_t2i","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/flickr30.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/flickr30.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"385a40328778e8cb"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/model.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"0b9ff430cf687b70","entry":"get_clip_metrics","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/dci_long.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/dci_long.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b9ff430cf687b70"}},{"code_sha256_prefix":"0434c3049ad999d2","entry":"get_image_feature","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/flickr30.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/flickr30.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0434c3049ad999d2"}},{"code_sha256_prefix":"b476c8cfbf0f1f47","entry":"get_input_dtype","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/model.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b476c8cfbf0f1f47"}},{"code_sha256_prefix":"93448c613c050973","entry":"get_latest_checkpoint","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval_tulip.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval_tulip.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93448c613c050973"}},{"code_sha256_prefix":"5be654577f88529b","entry":"get_text_feature","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/flickr30.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/flickr30.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5be654577f88529b"}},{"code_sha256_prefix":"287dbd8a0930623d","entry":"run_urban1k_openclip","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/urban1k.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/urban1k.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"287dbd8a0930623d"}},{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/loss.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"e332856e3c2fc814","entry":"neighbour_exchange","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/loss.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e332856e3c2fc814"}},{"code_sha256_prefix":"5b1fd364afcf3c05","entry":"neighbour_exchange_bidir","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/loss.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b1fd364afcf3c05"}},{"code_sha256_prefix":"fb651d0a97fd4d3f","entry":"prepare_inputs_for_generation","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/coca_model.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/coca_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb651d0a97fd4d3f"}},{"code_sha256_prefix":"2a377da4a76a2d44","entry":"register_pooler","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"open_clip/hf_model.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/open_clip/hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a377da4a76a2d44"}},{"code_sha256_prefix":"fef28a6f8d2d2a7a","entry":"run_coco","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/coco.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/coco.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fef28a6f8d2d2a7a"}},{"code_sha256_prefix":"4a7520e4ce7edf46","entry":"run_dci_long_openclip","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/dci_long.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/dci_long.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a7520e4ce7edf46"}},{"code_sha256_prefix":"365d03d6a9cc44e6","entry":"run_sharegpt4v","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/sharegpt4v.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/sharegpt4v.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"365d03d6a9cc44e6"}},{"code_sha256_prefix":"388875d0199845b5","entry":"run_sharegpt4v_openclip","repo":"ivonajdenkoska/tulip","repo_kind":"official","path":"eval/sharegpt4v.py","file_url":"https://github.com/ivonajdenkoska/tulip/blob/HEAD/eval/sharegpt4v.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"388875d0199845b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}