{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-fourier-prompt-tuning","title":"Visual Fourier Prompt Tuning","arxiv_id":"2411.01327","date":"2024-11-02","proceeding":null,"authors":["Runjia Zeng","Cheng Han","Qifan Wang","Chunshu Wu","Tong Geng","Lifu Huang","Ying Nian Wu","Dongfang Liu"],"abstract":"With the scale of vision Transformer-based models continuing to grow, finetuning these large-scale pretrained models for new tasks has become increasingly parameter-intensive. Visual prompt tuning is introduced as a parameter-efficient finetuning (PEFT) method to this trend. Despite its successes, a notable research challenge persists within almost all PEFT approaches: significant performance degradation is observed when there is a substantial disparity between the datasets applied in pretraining and finetuning phases. To address this challenge, we draw inspiration from human visual cognition, and propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach innovatively incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Apart from its inherent simplicity and intuitiveness, VFPT exhibits superior performance across all datasets, offering a general solution to dataset challenges, irrespective of data disparities. Empirical results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks, with low parameter usage (e.g., 0.57% of model parameters on VTAB-1k) and notable performance enhancements (e.g., 73.20% of mean accuracy on VTAB-1k). Our code is avaliable at https://github.com/runtsang/VFPT.","url_abs":"https://arxiv.org/abs/2411.01327v2","url_pdf":"https://arxiv.org/pdf/2411.01327v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-fourier-prompt-tuning","repo_url":"https://github.com/runtsang/vfpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"visual-prompt-tuning","task_name":"Visual Prompt Tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.01327","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.01327"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/runtsang/VFPT","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/runtsang/vfpt","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_violates":1,"ran_honours":1,"ran_fixture":1,"unverified":14},"by_repo_kind":{"official":{"samples":17,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"94402f271c35bec0","entry":"gray2rgb","repo":"runtsang/vfpt","repo_kind":"official","path":"src/models/vit_prompt/vit_fourier.py","file_url":"https://github.com/runtsang/vfpt/blob/HEAD/src/models/vit_prompt/vit_fourier.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"94402f271c35bec0"}},{"code_sha256_prefix":"b03459906b2b6a03","entry":"calc_spectrum","repo":"runtsang/vfpt","repo_kind":"official","path":"src/models/vit_prompt/vit_fourier.py","file_url":"https://github.com/runtsang/vfpt/blob/HEAD/src/models/vit_prompt/vit_fourier.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b03459906b2b6a03"}},{"code_sha256_prefix":"3fb8103712439145","entry":"plot_spectrum","repo":"runtsang/vfpt","repo_kind":"official","path":"src/models/vit_prompt/vit_fourier.py","file_url":"https://github.com/runtsang/vfpt/blob/HEAD/src/models/vit_prompt/vit_fourier.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3fb8103712439145"}},{"code_sha256_prefix":"3e009472c0d2fa50","entry":"find_best_lrwd","repo":"runtsang/VFPT","repo_kind":"official","path":"tune_fgvc_final.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/tune_fgvc_final.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3e009472c0d2fa50"}},{"code_sha256_prefix":"b71d9b398f930b09","entry":"get_job_indices","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_scheduler.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b71d9b398f930b09"}},{"code_sha256_prefix":"7ead1a7c9a99873f","entry":"get_random_states","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_net_plotter.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_net_plotter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7ead1a7c9a99873f"}},{"code_sha256_prefix":"3e80953c5e1e0841","entry":"get_random_weights","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_net_plotter.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_net_plotter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3e80953c5e1e0841"}},{"code_sha256_prefix":"cd4ae422bd97483a","entry":"get_unplotted_indices","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_scheduler.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cd4ae422bd97483a"}},{"code_sha256_prefix":"5fa87577ccee9cca","entry":"get_weights","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_net_plotter.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_net_plotter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5fa87577ccee9cca"}},{"code_sha256_prefix":"e33e2e1faf24f6b7","entry":"is_master","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_mpi4pytorch.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_mpi4pytorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e33e2e1faf24f6b7"}},{"code_sha256_prefix":"e29de9d44f616daa","entry":"name_surface_file","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_plot_surface.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_plot_surface.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e29de9d44f616daa"}},{"code_sha256_prefix":"c2cdd6e1a4874b83","entry":"nplist_to_tensor","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_projection.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_projection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c2cdd6e1a4874b83"}},{"code_sha256_prefix":"6f49307171759669","entry":"npvec_to_tensorlist","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_projection.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_projection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6f49307171759669"}},{"code_sha256_prefix":"0d11ac9cea6805a3","entry":"read_list","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_h5_util.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_h5_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0d11ac9cea6805a3"}},{"code_sha256_prefix":"3191424d76122507","entry":"setup_surface_file","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_plot_surface.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_plot_surface.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3191424d76122507"}},{"code_sha256_prefix":"5e0c7487c2e7cb09","entry":"split_inds","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_scheduler.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5e0c7487c2e7cb09"}},{"code_sha256_prefix":"28fefee105aa3460","entry":"tensorlist_to_tensor","repo":"runtsang/VFPT","repo_kind":"official","path":"ls_projection.py","file_url":"https://github.com/runtsang/VFPT/blob/HEAD/ls_projection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"28fefee105aa3460"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}