{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-and-mitigating-miscalibration","title":"Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models","arxiv_id":"2410.02681","date":"2024-10-03","proceeding":null,"authors":["Shuoyuan Wang","Yixuan Li","Hongxin Wei"],"abstract":"Confidence calibration is critical for the safe deployment of machine learning models in the real world. However, such issue in vision-language models like CLIP, particularly after fine-tuning, has not been fully addressed. In this work, we demonstrate that existing prompt tuning methods usually lead to a trade-off of calibration between base and new classes: the cross-entropy loss in CoOp causes overconfidence in new classes by increasing textual label divergence, whereas the regularization of KgCoOp maintains the confidence level but results in underconfidence in base classes due to the improved accuracy. Inspired by the observations, we introduce Dynamic Outlier Regularization (DOR) to ensure the confidence calibration on both base and new classes after fine-tuning. In particular, we propose to minimize the feature deviation of novel textual labels (instead of base classes) sampled from a large vocabulary. In effect, DOR prevents the increase in textual divergence for new labels while easing restrictions on base classes. Extensive experiments demonstrate that DOR can enhance the calibration performance of current fine-tuning methods on base and new classes.","url_abs":"https://arxiv.org/abs/2410.02681v1","url_pdf":"https://arxiv.org/pdf/2410.02681v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-and-mitigating-miscalibration","repo_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"coop","method_name":"CoOp"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.02681","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02681"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ml-stat-Sustech/Outlier-Calibration","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":1,"unverified":5},"by_repo_kind":{"listed":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip/simple_tokenizer.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"ba26afd892405335","entry":"compute_ci95","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"parse_test_res.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/parse_test_res.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba26afd892405335"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip/simple_tokenizer.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip/simple_tokenizer.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"c47aa9e5b049a11d","entry":"build_model","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip_coprompt/model_clip_original.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip_coprompt/model_clip_original.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c47aa9e5b049a11d"}},{"code_sha256_prefix":"918dfac560aed1c9","entry":"build_model","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip_coprompt/model.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip_coprompt/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"918dfac560aed1c9"}},{"code_sha256_prefix":"fd9041fbaf4e48a4","entry":"build_model","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip/model.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd9041fbaf4e48a4"}},{"code_sha256_prefix":"fbf8c0143d9c48e3","entry":"load","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip_coprompt/clip.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip_coprompt/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbf8c0143d9c48e3"}},{"code_sha256_prefix":"c22a78c25372b59f","entry":"load","repo":"ml-stat-Sustech/Outlier-Calibration","repo_kind":"listed","path":"clip/clip.py","file_url":"https://github.com/ml-stat-Sustech/Outlier-Calibration/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c22a78c25372b59f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}