{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sharpzo-hybrid-sharpness-aware-vision","title":"SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes","arxiv_id":"2506.20990","date":"2025-06-26","proceeding":null,"authors":["Yifan Yang","Zhen Zhang","Rupak Vignesh Swaminathan","Jing Liu","Nathan Susanj","Zheng Zhang"],"abstract":"Fine-tuning vision language models (VLMs) has achieved remarkable performance across various downstream tasks; yet, it requires access to model gradients through backpropagation (BP), making them unsuitable for memory-constrained, inference-only edge devices. To address this limitation, previous work has explored various BP-free fine-tuning methods. However, these approaches often rely on high-variance evolutionary strategies (ES) or zeroth-order (ZO) optimization, and often fail to achieve satisfactory performance. In this paper, we propose a hybrid Sharpness-aware Zeroth-order optimization (SharpZO) approach, specifically designed to enhance the performance of ZO VLM fine-tuning via a sharpness-aware warm-up training. SharpZO features a two-stage optimization process: a sharpness-aware ES stage that globally explores and smooths the loss landscape to construct a strong initialization, followed by a fine-grained local search via sparse ZO optimization. The entire optimization relies solely on forward passes. Detailed theoretical analysis and extensive experiments on CLIP models demonstrate that SharpZO significantly improves accuracy and convergence speed, achieving up to 7% average gain over state-of-the-art forward-only methods.","url_abs":"https://arxiv.org/abs/2506.20990v1","url_pdf":"https://arxiv.org/pdf/2506.20990v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sharpzo-hybrid-sharpness-aware-vision","repo_url":"https://github.com/yifanycc/sharpzo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.20990","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.20990"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yifanycc/sharpzo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"yifanycc/sharpzo","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"199d6f35dcbfe379","entry":"ce_loss","repo":"yifanycc/sharpzo","repo_kind":"official","path":"clip/bb_clip_model.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/clip/bb_clip_model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"199d6f35dcbfe379"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"yifanycc/sharpzo","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"yifanycc/sharpzo","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"5b58ecca1df5fb53","entry":"build_model","repo":"yifanycc/sharpzo","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b58ecca1df5fb53"}},{"code_sha256_prefix":"3d3bb1e71ff2f61e","entry":"ce_loss","repo":"yifanycc/sharpzo","repo_kind":"official","path":"main_sharpzo.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/main_sharpzo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d3bb1e71ff2f61e"}},{"code_sha256_prefix":"f2a0a5061e7c204e","entry":"cls_acc","repo":"yifanycc/sharpzo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2a0a5061e7c204e"}},{"code_sha256_prefix":"fe317c3d4d24147a","entry":"listdir_nohidden","repo":"yifanycc/sharpzo","repo_kind":"official","path":"datasets/imagenet.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/datasets/imagenet.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe317c3d4d24147a"}},{"code_sha256_prefix":"db64af1ba3479e8d","entry":"load","repo":"yifanycc/sharpzo","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db64af1ba3479e8d"}},{"code_sha256_prefix":"f539d9cc7267493c","entry":"pre_load_features","repo":"yifanycc/sharpzo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yifanycc/sharpzo/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f539d9cc7267493c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}