{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-calibrated-robust-fine-tuning-of","title":"Towards Calibrated Robust Fine-Tuning of Vision-Language Models","arxiv_id":"2311.01723","date":"2023-11-03","proceeding":null,"authors":["Changdae Oh","Hyesu Lim","Mijoo Kim","Dongyoon Han","Sangdoo Yun","Jaegul Choo","Alexander Hauptmann","Zhi-Qi Cheng","Kyungwoo Song"],"abstract":"Improving out-of-distribution (OOD) generalization during in-distribution (ID) adaptation is a primary goal of robust fine-tuning of zero-shot models beyond naive fine-tuning. However, despite decent OOD generalization performance from recent robust fine-tuning methods, confidence calibration for reliable model output has not been fully addressed. This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. Firstly, we show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data: 1) ID calibration error and 2) the smallest singular value of the ID input covariance matrix. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value, which is further guided by the self-distillation of a moving-averaged model to achieve calibrated prediction as well. Starting from empirical evidence supporting our theoretical statements, we provide extensive experimental results on ImageNet distribution shift benchmarks that demonstrate the effectiveness of our theorem and its practical implementation.","url_abs":"https://arxiv.org/abs/2311.01723v7","url_pdf":"https://arxiv.org/pdf/2311.01723v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-calibrated-robust-fine-tuning-of","repo_url":"https://github.com/MLAI-Yonsei/CaRot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"medical-diagnosis","task_name":"Medical Diagnosis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.01723","atlas_url":"https://app.syntology.ai/?focus=2311.01723","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.01723"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MLAI-Yonsei/CaRot","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":4,"ran":1,"ran_fixture":1,"unverified":6},"by_repo_kind":{"official":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"clip/tokenizer.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"6e97baf23bf18650","entry":"compute_calibration","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"src/visualize.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/src/visualize.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e97baf23bf18650"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"clip/tokenizer.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"b1e9e0d3d7d7e615","entry":"maybe_dictionarize","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"src/datasets_/common.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/src/datasets_/common.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1e9e0d3d7d7e615"}},{"code_sha256_prefix":"9fbea808db47dc66","entry":"project_logits","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"src/datasets_/imagenet.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/src/datasets_/imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fbea808db47dc66"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"clip/tokenizer.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"b64f67bec6a093b9","entry":"build_model","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b64f67bec6a093b9"}},{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"clip/loss.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"35ec3a8e556cc61c","entry":"get_features","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"src/datasets_/common.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/src/datasets_/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35ec3a8e556cc61c"}},{"code_sha256_prefix":"74edffa1dfc39e58","entry":"get_features_helper","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"src/datasets_/common.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/src/datasets_/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74edffa1dfc39e58"}},{"code_sha256_prefix":"d558de6dd71f4220","entry":"reliability_diagram","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"src/visualize.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/src/visualize.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d558de6dd71f4220"}},{"code_sha256_prefix":"435288e8d2fd445a","entry":"reliability_diagrams","repo":"MLAI-Yonsei/CaRot","repo_kind":"official","path":"src/visualize.py","file_url":"https://github.com/MLAI-Yonsei/CaRot/blob/HEAD/src/visualize.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"435288e8d2fd445a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}