{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fine-tuning-is-fine-if-calibrated","title":"Fine-Tuning is Fine, if Calibrated","arxiv_id":"2409.16223","date":"2024-09-24","proceeding":null,"authors":["Zheda Mai","Arpita Chowdhury","Ping Zhang","Cheng-Hao Tu","Hong-You Chen","Vardaan Pahuja","Tanya Berger-Wolf","Song Gao","Charles Stewart","Yu Su","Wei-Lun Chao"],"abstract":"Fine-tuning is arguably the most straightforward way to tailor a pre-trained model (e.g., a foundation model) to downstream applications, but it also comes with the risk of losing valuable knowledge the model had learned in pre-training. For example, fine-tuning a pre-trained classifier capable of recognizing a large number of classes to master a subset of classes at hand is shown to drastically degrade the model's accuracy in the other classes it had previously learned. As such, it is hard to further use the fine-tuned model when it encounters classes beyond the fine-tuning data. In this paper, we systematically dissect the issue, aiming to answer the fundamental question, \"What has been damaged in the fine-tuned model?\" To our surprise, we find that the fine-tuned model neither forgets the relationship among the other classes nor degrades the features to recognize these classes. Instead, the fine-tuned model often produces more discriminative features for these other classes, even if they were missing during fine-tuning! {What really hurts the accuracy is the discrepant logit scales between the fine-tuning classes and the other classes}, implying that a simple post-processing calibration would bring back the pre-trained model's capability and at the same time unveil the feature improvement over all classes. We conduct an extensive empirical study to demonstrate the robustness of our findings and provide preliminary explanations underlying them, suggesting new directions for future theoretical analysis. Our code is available at https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated.","url_abs":"https://arxiv.org/abs/2409.16223v3","url_pdf":"https://arxiv.org/pdf/2409.16223v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fine-tuning-is-fine-if-calibrated","repo_url":"https://github.com/osu-mlb/fine-tuning-is-fine-if-calibrated","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.16223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.16223"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/osu-mlb/fine-tuning-is-fine-if-calibrated","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"unverified":6},"by_repo_kind":{"official":{"samples":15,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d3790fd60c5077a","entry":"all_in","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/common.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d3790fd60c5077a"}},{"code_sha256_prefix":"654b3d00bbf1c923","entry":"dtn","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/model/_digits.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/model/_digits.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"654b3d00bbf1c923"}},{"code_sha256_prefix":"1d66cc48b63fca52","entry":"flatten_list","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/common.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d66cc48b63fca52"}},{"code_sha256_prefix":"1f88ab8c26c03b73","entry":"get_class_mean","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/evaluate.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f88ab8c26c03b73"}},{"code_sha256_prefix":"d829c6631f3d92c6","entry":"get_parameters","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/model.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d829c6631f3d92c6"}},{"code_sha256_prefix":"70367b6dda994032","entry":"is_json","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/common.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70367b6dda994032"}},{"code_sha256_prefix":"87c67b9512a15d16","entry":"lenet","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/model/_digits.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/model/_digits.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87c67b9512a15d16"}},{"code_sha256_prefix":"095e3291b31a3a1e","entry":"mask_score","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/evaluate.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"095e3291b31a3a1e"}},{"code_sha256_prefix":"b2ae4997a88217a1","entry":"resnet34_ibn_a","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/model/_ibn_models.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/model/_ibn_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2ae4997a88217a1"}},{"code_sha256_prefix":"8b7318d02fdd57f5","entry":"GET_HARDCODED_CLASSES","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/HT_HARDCODED.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/HT_HARDCODED.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b7318d02fdd57f5"}},{"code_sha256_prefix":"15b6ce87cea6a1a2","entry":"create_model","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/model.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15b6ce87cea6a1a2"}},{"code_sha256_prefix":"619286e8e6e8c6ad","entry":"get_model","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/model.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"619286e8e6e8c6ad"}},{"code_sha256_prefix":"a95000b4197680ef","entry":"mask_predict_score","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/util/evaluate.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/util/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a95000b4197680ef"}},{"code_sha256_prefix":"c7b47196a0364d1b","entry":"resnet18_ibn_a","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/model/_ibn_models.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/model/_ibn_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7b47196a0364d1b"}},{"code_sha256_prefix":"3a805a0c566b4de7","entry":"resnet50_ibn_a","repo":"OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated","repo_kind":"official","path":"py/model/_ibn_models.py","file_url":"https://github.com/OSU-MLB/Fine-Tuning-Is-Fine-If-Calibrated/blob/HEAD/py/model/_ibn_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a805a0c566b4de7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}