{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enhancing-zero-shot-vision-models-by-label","title":"Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting","arxiv_id":"2410.19294","date":"2024-10-25","proceeding":null,"authors":["Xingyu Zhu","Beier Zhu","Yi Tan","Shuo Wang","Yanbin Hao","Hanwang Zhang"],"abstract":"Vision-language models, such as CLIP, have shown impressive generalization capacities when using appropriate text descriptions. While optimizing prompts on downstream labeled data has proven effective in improving performance, these methods entail labor costs for annotations and are limited by their quality. Additionally, since CLIP is pre-trained on highly imbalanced Web-scale data, it suffers from inherent label bias that leads to suboptimal performance. To tackle the above challenges, we propose a label-Free prompt distribution learning and bias correction framework, dubbed as **Frolic**, which boosts zero-shot performance without the need for labeled data. Specifically, our Frolic learns distributions over prompt prototypes to capture diverse visual representations and adaptively fuses these with the original CLIP through confidence matching. This fused model is further enhanced by correcting label bias via a label-free logit adjustment. Notably, our method is not only training-free but also circumvents the necessity for hyper-parameter tuning. Extensive experimental results across 16 datasets demonstrate the efficacy of our approach, particularly outperforming the state-of-the-art by an average of $2.6\\%$ on 10 datasets with CLIP ViT-B/16 and achieving an average margin of $1.5\\%$ on ImageNet and its five distribution shifts with CLIP ViT-B/16. Codes are available in https://github.com/zhuhsingyuu/Frolic.","url_abs":"https://arxiv.org/abs/2410.19294v1","url_pdf":"https://arxiv.org/pdf/2410.19294v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.19294","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.19294"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zhuhsingyuu/Frolic","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"589b2c2591399402","entry":"power_iteration","repo":"zhuhsingyuu/Frolic","repo_kind":"found_in_text","path":"main_zeroshot.py","file_url":"https://github.com/zhuhsingyuu/Frolic/blob/HEAD/main_zeroshot.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"589b2c2591399402"}},{"code_sha256_prefix":"142e428b4d605144","entry":"pp_estimate_eigen","repo":"zhuhsingyuu/Frolic","repo_kind":"found_in_text","path":"main_zeroshot.py","file_url":"https://github.com/zhuhsingyuu/Frolic/blob/HEAD/main_zeroshot.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"142e428b4d605144"}},{"code_sha256_prefix":"a32e53753e69a604","entry":"run","repo":"zhuhsingyuu/Frolic","repo_kind":"found_in_text","path":"main_zeroshot.py","file_url":"https://github.com/zhuhsingyuu/Frolic/blob/HEAD/main_zeroshot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a32e53753e69a604"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}