{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-visual-prompt-tuning-by-gaussian","title":"Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition","arxiv_id":"2410.21042","date":"2024-10-28","proceeding":null,"authors":["Mengke Li","Ye Liu","Yang Lu","Yiqun Zhang","Yiu-ming Cheung","Hui Huang"],"abstract":"Long-tail learning has garnered widespread attention and achieved significant progress in recent times. However, even with pre-trained prior knowledge, models still exhibit weaker generalization performance on tail classes. The promising Sharpness-Aware Minimization (SAM) can effectively improve the generalization capability of models by seeking out flat minima in the loss landscape, which, however, comes at the cost of doubling the computational time. Since the update rule of SAM necessitates two consecutive (non-parallelizable) forward and backpropagation at each step. To address this issue, we propose a novel method called Random SAM prompt tuning (RSAM-PT) to improve the model generalization, requiring only one-step gradient computation at each step. Specifically, we search for the gradient descent direction within a random neighborhood of the parameters during each gradient update. To amplify the impact of tail-class samples and avoid overfitting, we employ the deferred re-weight scheme to increase the significance of tail-class samples. The classification accuracy of long-tailed data can be significantly improved by the proposed RSAM-PT, particularly for tail classes. RSAM-PT achieves the state-of-the-art performance of 90.3\\%, 76.5\\%, and 50.1\\% on benchmark datasets CIFAR100-LT (IF 100), iNaturalist 2018, and Places-LT, respectively. The source code is temporarily available at https://github.com/Keke921/GNM-PT.","url_abs":"https://arxiv.org/abs/2410.21042v1","url_pdf":"https://arxiv.org/pdf/2410.21042v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-visual-prompt-tuning-by-gaussian","repo_url":"https://github.com/keke921/gnm-pt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"improving-visual-prompt-tuning-by-gaussian","repo_url":"https://github.com/keke921/gclloss","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"improving-visual-prompt-tuning-by-gaussian","repo_url":"https://github.com/keke921/h2t","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"long-tail-learning","task_name":"Long-tail Learning"},{"task_slug":"visual-prompt-tuning","task_name":"Visual Prompt Tuning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"sam","method_name":"SAM"},{"method_slug":"sharpness-aware-minimization","method_name":"Sharpness-Aware Minimization"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.21042","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.21042"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Keke921/GNM-PT","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keke921/gnm-pt","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keke921/gclloss","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keke921/h2t","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1b3f82e6c53766e7","entry":"focal_loss","repo":"Keke921/GNM-PT","repo_kind":"official","path":"loss.py","file_url":"https://github.com/Keke921/GNM-PT/blob/HEAD/loss.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b3f82e6c53766e7"}},{"code_sha256_prefix":"8539c33350598089","entry":"GNM","repo":"keke921/gnm-pt","repo_kind":"official","path":"gnm.py","file_url":"https://github.com/keke921/gnm-pt/blob/HEAD/gnm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8539c33350598089"}},{"code_sha256_prefix":"b51354bfb2a4f8ea","entry":"build_promptmodel","repo":"Keke921/GNM-PT","repo_kind":"official","path":"PromptModels/GetPromptModel.py","file_url":"https://github.com/Keke921/GNM-PT/blob/HEAD/PromptModels/GetPromptModel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b51354bfb2a4f8ea"}},{"code_sha256_prefix":"5dac11fdc41c896b","entry":"checkpoint_filter_fn","repo":"Keke921/GNM-PT","repo_kind":"official","path":"PromptModels/vit.py","file_url":"https://github.com/Keke921/GNM-PT/blob/HEAD/PromptModels/vit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dac11fdc41c896b"}},{"code_sha256_prefix":"1dbef2d4da4ee10f","entry":"resize_pos_embed","repo":"Keke921/GNM-PT","repo_kind":"official","path":"PromptModels/vit.py","file_url":"https://github.com/Keke921/GNM-PT/blob/HEAD/PromptModels/vit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dbef2d4da4ee10f"}},{"code_sha256_prefix":"5a7b1c8f58adefd0","entry":"train","repo":"keke921/gnm-pt","repo_kind":"official","path":"vtab/train_full.py","file_url":"https://github.com/keke921/gnm-pt/blob/HEAD/vtab/train_full.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a7b1c8f58adefd0"}},{"code_sha256_prefix":"8a2d99dd6fd9f0de","entry":"train","repo":"Keke921/GNM-PT","repo_kind":"official","path":"train_full-lmk-st2.py","file_url":"https://github.com/Keke921/GNM-PT/blob/HEAD/train_full-lmk-st2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a2d99dd6fd9f0de"}},{"code_sha256_prefix":"45f78a4d8810cd4d","entry":"validate","repo":"keke921/gnm-pt","repo_kind":"official","path":"vtab/train_full.py","file_url":"https://github.com/keke921/gnm-pt/blob/HEAD/vtab/train_full.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45f78a4d8810cd4d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}