{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nemesis-normalizing-the-soft-prompt-vectors","title":"Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models","arxiv_id":"2408.13979","date":"2024-08-26","proceeding":null,"authors":["Shuai Fu","Xiequn Wang","Qiushi Huang","Yu Zhang"],"abstract":"With the prevalence of large-scale pretrained vision-language models (VLMs), such as CLIP, soft-prompt tuning has become a popular method for adapting these models to various downstream tasks. However, few works delve into the inherent properties of learnable soft-prompt vectors, specifically the impact of their norms to the performance of VLMs. This motivates us to pose an unexplored research question: ``Do we need to normalize the soft prompts in VLMs?'' To fill this research gap, we first uncover a phenomenon, called the \\textbf{Low-Norm Effect} by performing extensive corruption experiments, suggesting that reducing the norms of certain learned prompts occasionally enhances the performance of VLMs, while increasing them often degrades it. To harness this effect, we propose a novel method named \\textbf{N}ormalizing th\\textbf{e} soft-pro\\textbf{m}pt v\\textbf{e}ctors of vi\\textbf{si}on-language model\\textbf{s} (\\textbf{Nemesis}) to normalize soft-prompt vectors in VLMs. To the best of our knowledge, our work is the first to systematically investigate the role of norms of soft-prompt vector in VLMs, offering valuable insights for future research in soft-prompt tuning. The code is available at \\texttt{\\href{https://github.com/ShyFoo/Nemesis}{https://github.com/ShyFoo/Nemesis}}.","url_abs":"https://arxiv.org/abs/2408.13979v1","url_pdf":"https://arxiv.org/pdf/2408.13979v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nemesis-normalizing-the-soft-prompt-vectors","repo_url":"https://github.com/shyfoo/nemesis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.13979","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.13979"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ShyFoo/Nemesis","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shyfoo/nemesis","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"ShyFoo/Nemesis","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/ShyFoo/Nemesis/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"ShyFoo/Nemesis","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/ShyFoo/Nemesis/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"ShyFoo/Nemesis","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/ShyFoo/Nemesis/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"c47aa9e5b049a11d","entry":"build_model","repo":"ShyFoo/Nemesis","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/ShyFoo/Nemesis/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c47aa9e5b049a11d"}},{"code_sha256_prefix":"4036f90d62ffe676","entry":"build_model","repo":"ShyFoo/Nemesis","repo_kind":"official","path":"clip_plot/model.py","file_url":"https://github.com/ShyFoo/Nemesis/blob/HEAD/clip_plot/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4036f90d62ffe676"}},{"code_sha256_prefix":"fbf8c0143d9c48e3","entry":"load","repo":"ShyFoo/Nemesis","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/ShyFoo/Nemesis/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbf8c0143d9c48e3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}