{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-visual-prompt-tuning-for-self","title":"Improving Visual Prompt Tuning for Self-supervised Vision Transformers","arxiv_id":"2306.05067","date":"2023-06-08","proceeding":null,"authors":["Seungryong Yoo","Eunji Kim","Dahuin Jung","Jungbeom Lee","Sungroh Yoon"],"abstract":"Visual Prompt Tuning (VPT) is an effective tuning method for adapting pretrained Vision Transformers (ViTs) to downstream tasks. It leverages extra learnable tokens, known as prompts, which steer the frozen pretrained ViTs. Although VPT has demonstrated its applicability with supervised vision transformers, it often underperforms with self-supervised ones. Through empirical observations, we deduce that the effectiveness of VPT hinges largely on the ViT blocks with which the prompt tokens interact. Specifically, VPT shows improved performance on image classification tasks for MAE and MoCo v3 when the prompt tokens are inserted into later blocks rather than the first block. These observations suggest that there exists an optimal location of blocks for the insertion of prompt tokens. Unfortunately, identifying the optimal blocks for prompts within each self-supervised ViT for diverse future scenarios is a costly process. To mitigate this problem, we propose a simple yet effective method that learns a gate for each ViT block to adjust its intervention into the prompt tokens. With our method, prompt tokens are selectively influenced by blocks that require steering for task adaptation. Our method outperforms VPT variants in FGVC and VTAB image classification and ADE20K semantic segmentation. The code is available at https://github.com/ryongithub/GatedPromptTuning.","url_abs":"https://arxiv.org/abs/2306.05067v1","url_pdf":"https://arxiv.org/pdf/2306.05067v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-visual-prompt-tuning-for-self","repo_url":"https://github.com/ryongithub/gatedprompttuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"visual-prompt-tuning","task_name":"Visual Prompt Tuning"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"infonce","method_name":"InfoNCE"},{"method_slug":"mae","method_name":"MAE"},{"method_slug":"moco","method_name":"MoCo"},{"method_slug":"moco-v3","method_name":"MoCo v3"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-prompt-tuning-on-fgvc","task":"Visual Prompt Tuning","dataset":"FGVC","model":"GateVPT(ViT-B/16_MoCo_v3_pretrained_ImageNet-1K)","rank_in_archive_order":5,"of":10,"metrics":{"Mean Accuracy":"83.00"},"uses_additional_data":false},{"leaderboard":"/sota/visual-prompt-tuning-on-fgvc","task":"Visual Prompt Tuning","dataset":"FGVC","model":"GateVPT(ViT-B/16_MAE_pretrained_ImageNet-1K)","rank_in_archive_order":8,"of":10,"metrics":{"Mean Accuracy":"73.39"},"uses_additional_data":false},{"leaderboard":"/sota/visual-prompt-tuning-on-vtab-1k-natural-7","task":"Visual Prompt Tuning","dataset":"VTAB-1k(Natural<7>)","model":"GateVPT(ViT-B/16_MoCo_v3_pretrained_ImageNet-1K)","rank_in_archive_order":2,"of":10,"metrics":{"Mean Accuracy":"74.84"},"uses_additional_data":false},{"leaderboard":"/sota/visual-prompt-tuning-on-vtab-1k-natural-7","task":"Visual Prompt Tuning","dataset":"VTAB-1k(Natural<7>)","model":"GateVPT(ViT-B/16_MAE_pretrained_ImageNet-1K)","rank_in_archive_order":8,"of":10,"metrics":{"Mean Accuracy":"47.61"},"uses_additional_data":false},{"leaderboard":"/sota/visual-prompt-tuning-on-vtab-1k-specialized-4","task":"Visual Prompt Tuning","dataset":"VTAB-1k(Specialized<4>)","model":"GateVPT(ViT-B/16_MoCo_v3_pretrained_ImageNet-1K)","rank_in_archive_order":3,"of":10,"metrics":{"Mean Accuracy":"83.38"},"uses_additional_data":false},{"leaderboard":"/sota/visual-prompt-tuning-on-vtab-1k-specialized-4","task":"Visual Prompt Tuning","dataset":"VTAB-1k(Specialized<4>)","model":"GateVPT(ViT-B/16_MAE_pretrained_ImageNet-1K)","rank_in_archive_order":8,"of":10,"metrics":{"Mean Accuracy":"76.86"},"uses_additional_data":false},{"leaderboard":"/sota/visual-prompt-tuning-on-vtab-1k-structured-8","task":"Visual Prompt Tuning","dataset":"VTAB-1k(Structured<8>)","model":"GateVPT(ViT-B/16_MoCo_v3_pretrained_ImageNet-1K)","rank_in_archive_order":5,"of":10,"metrics":{"Mean Accuracy":"49.10"},"uses_additional_data":false},{"leaderboard":"/sota/visual-prompt-tuning-on-vtab-1k-structured-8","task":"Visual Prompt Tuning","dataset":"VTAB-1k(Structured<8>)","model":"GateVPT(ViT-B/16_MAE_pretrained_ImageNet-1K)","rank_in_archive_order":8,"of":10,"metrics":{"Mean Accuracy":"36.80"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.05067","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.05067"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ryongithub/gatedprompttuning","reach":null}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9d53ec26dfcd7416","entry":"Attention","repo":"ryongithub/gatedprompttuning","repo_kind":"official","path":"src/models/vit_prompt/vit_mae.py","file_url":"https://github.com/ryongithub/gatedprompttuning/blob/HEAD/src/models/vit_prompt/vit_mae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d53ec26dfcd7416"}},{"code_sha256_prefix":"5667ba706eacf9a5","entry":"Block","repo":"ryongithub/gatedprompttuning","repo_kind":"official","path":"src/models/vit_prompt/vit_mae.py","file_url":"https://github.com/ryongithub/gatedprompttuning/blob/HEAD/src/models/vit_prompt/vit_mae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5667ba706eacf9a5"}},{"code_sha256_prefix":"f949bff696b4d036","entry":"PromptedVisionTransformer","repo":"ryongithub/gatedprompttuning","repo_kind":"official","path":"src/models/vit_prompt/vit_mae.py","file_url":"https://github.com/ryongithub/gatedprompttuning/blob/HEAD/src/models/vit_prompt/vit_mae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f949bff696b4d036"}},{"code_sha256_prefix":"33fb7be835804856","entry":"VisionTransformer","repo":"ryongithub/gatedprompttuning","repo_kind":"official","path":"src/models/vit_prompt/vit_mae.py","file_url":"https://github.com/ryongithub/gatedprompttuning/blob/HEAD/src/models/vit_prompt/vit_mae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33fb7be835804856"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}