{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sa-2-vp-spatially-aligned-and-adapted-visual","title":"SA$^2$VP: Spatially Aligned-and-Adapted Visual Prompt","arxiv_id":"2312.10376","date":"2023-12-16","proceeding":null,"authors":["Wenjie Pei","Tongqi Xia","Fanglin Chen","Jinsong Li","Jiandong Tian","Guangming Lu"],"abstract":"As a prominent parameter-efficient fine-tuning technique in NLP, prompt tuning is being explored its potential in computer vision. Typical methods for visual prompt tuning follow the sequential modeling paradigm stemming from NLP, which represents an input image as a flattened sequence of token embeddings and then learns a set of unordered parameterized tokens prefixed to the sequence representation as the visual prompts for task adaptation of large vision models. While such sequential modeling paradigm of visual prompt has shown great promise, there are two potential limitations. First, the learned visual prompts cannot model the underlying spatial relations in the input image, which is crucial for image encoding. Second, since all prompt tokens play the same role of prompting for all image tokens without distinction, it lacks the fine-grained prompting capability, i.e., individual prompting for different image tokens. In this work, we propose the \\mymodel model (\\emph{SA$^2$VP}), which learns a two-dimensional prompt token map with equal (or scaled) size to the image token map, thereby being able to spatially align with the image map. Each prompt token is designated to prompt knowledge only for the spatially corresponding image tokens. As a result, our model can conduct individual prompting for different image tokens in a fine-grained manner. Moreover, benefiting from the capability of preserving the spatial structure by the learned prompt token map, our \\emph{SA$^2$VP} is able to model the spatial relations in the input image, leading to more effective prompting. Extensive experiments on three challenging benchmarks for image classification demonstrate the superiority of our model over other state-of-the-art methods for visual prompt tuning. Code is available at \\emph{https://github.com/tommy-xq/SA2VP}.","url_abs":"https://arxiv.org/abs/2312.10376v1","url_pdf":"https://arxiv.org/pdf/2312.10376v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sa-2-vp-spatially-aligned-and-adapted-visual","repo_url":"https://github.com/tommy-xq/sa2vp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"visual-prompt-tuning","task_name":"Visual Prompt Tuning"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.10376","atlas_url":"https://app.syntology.ai/?focus=2312.10376","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.10376"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tommy-xq/sa2vp","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":2,"ran":6,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"7d8eaf5537f699e4","entry":"np2th","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7d8eaf5537f699e4"}},{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"7468079f7c4e2b5e","entry":"cosine_scheduler_epoch","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7468079f7c4e2b5e"}},{"code_sha256_prefix":"a3448702d6025ce6","entry":"get_num_layer_for_vit","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"optim_factory.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/optim_factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3448702d6025ce6"}},{"code_sha256_prefix":"18ae3e6ef2cf02ec","entry":"has_file_allowed_extension","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"dataset_folder.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/dataset_folder.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18ae3e6ef2cf02ec"}},{"code_sha256_prefix":"8ac50ae8773b9431","entry":"is_image_file","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"dataset_folder.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/dataset_folder.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ac50ae8773b9431"}},{"code_sha256_prefix":"fb6aea1653b20bba","entry":"make_dataset","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"dataset_folder.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/dataset_folder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb6aea1653b20bba"}},{"code_sha256_prefix":"d15b621dd834bc4f","entry":"train_class_batch","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"engine_for_train.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/engine_for_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d15b621dd834bc4f"}},{"code_sha256_prefix":"f9fd6241d935f07b","entry":"window_partition","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f9fd6241d935f07b"}},{"code_sha256_prefix":"fb32094c6dbece71","entry":"window_reverse","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"fb32094c6dbece71"}},{"code_sha256_prefix":"b700fb0a3496b255","entry":"cosine_scheduler","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b700fb0a3496b255"}},{"code_sha256_prefix":"3e009472c0d2fa50","entry":"find_best_lrwd","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"vpt_main/tune_vtab.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/vpt_main/tune_vtab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e009472c0d2fa50"}},{"code_sha256_prefix":"29416043c7035e7c","entry":"get_loss_scale_for_deepspeed","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"engine_for_train.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/engine_for_train.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29416043c7035e7c"}},{"code_sha256_prefix":"0ef0019514eaaa09","entry":"get_parameter_groups","repo":"tommy-xq/sa2vp","repo_kind":"official","path":"optim_factory.py","file_url":"https://github.com/tommy-xq/sa2vp/blob/HEAD/optim_factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ef0019514eaaa09"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}