{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cpt-colorful-prompt-tuning-for-pre-trained","title":"CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models","arxiv_id":"2109.11797","date":"2021-09-24","proceeding":null,"authors":["Yuan YAO","Ao Zhang","Zhengyan Zhang","Zhiyuan Liu","Tat-Seng Chua","Maosong Sun"],"abstract":"Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists a significant gap between the objective forms of model pre-training and fine-tuning, resulting in a need for large amounts of labeled data to stimulate the visual grounding capability of VL-PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin (e.g., 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/CPT.","url_abs":"https://arxiv.org/abs/2109.11797v3","url_pdf":"https://arxiv.org/pdf/2109.11797v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cpt-colorful-prompt-tuning-for-pre-trained","repo_url":"https://github.com/thunlp/cpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"cpt-colorful-prompt-tuning-for-pre-trained","repo_url":"https://github.com/lunaproject22/rpa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.11797","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.11797"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/cpt","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lunaproject22/rpa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_draft_wrong":2,"unverified":10},"by_repo_kind":{"official":{"samples":13,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"89bdd6decba2b415","entry":"compute_score_with_logits","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/gqa_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/gqa_cpt.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89bdd6decba2b415"}},{"code_sha256_prefix":"e36ec4e0a52c6a4b","entry":"instance_bce_with_logits","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/gqa_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/gqa_cpt.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e36ec4e0a52c6a4b"}},{"code_sha256_prefix":"12516cd21b436092","entry":"trim_batch","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/gqa_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/gqa_cpt.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12516cd21b436092"}},{"code_sha256_prefix":"01da88ba6172509e","entry":"get_evaluate_file","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/refcoco_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/refcoco_cpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01da88ba6172509e"}},{"code_sha256_prefix":"e040a3d039f5af10","entry":"get_evaluate_method","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/refcoco_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/refcoco_cpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e040a3d039f5af10"}},{"code_sha256_prefix":"5ddea353591c2338","entry":"restore_training_settings","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/refcoco_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/refcoco_cpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ddea353591c2338"}},{"code_sha256_prefix":"d48bb07bc01dfdb7","entry":"template1","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/datasets/refcoco_zsl_cpt_dataset.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/datasets/refcoco_zsl_cpt_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d48bb07bc01dfdb7"}},{"code_sha256_prefix":"c0fd672552e0160f","entry":"template2","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/datasets/refcoco_zsl_cpt_dataset.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/datasets/refcoco_zsl_cpt_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0fd672552e0160f"}},{"code_sha256_prefix":"39e9626ebfd96e62","entry":"template3","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/datasets/refcoco_zsl_cpt_dataset.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/datasets/refcoco_zsl_cpt_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39e9626ebfd96e62"}},{"code_sha256_prefix":"7ab756d02fb91e21","entry":"tokenize","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/datasets/refcoco_fsl_cpt_dataset.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/datasets/refcoco_fsl_cpt_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ab756d02fb91e21"}},{"code_sha256_prefix":"0a1867725d8d31b1","entry":"tokenize","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/datasets/vg_cpt_dataset.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/datasets/vg_cpt_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a1867725d8d31b1"}},{"code_sha256_prefix":"2e1b1ca48e16a324","entry":"vcr_collate_fn","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/vcr_nsp_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/vcr_nsp_cpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e1b1ca48e16a324"}},{"code_sha256_prefix":"ffe4043326409d20","entry":"vcr_collate_fn","repo":"thunlp/cpt","repo_kind":"official","path":"Oscar/oscar/fewshot/vcr_qar_nsp_cpt.py","file_url":"https://github.com/thunlp/cpt/blob/HEAD/Oscar/oscar/fewshot/vcr_qar_nsp_cpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffe4043326409d20"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}