{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-29380","title":"TRACER: Persistent Regularization for Robust Multimodal Finetuning","arxiv_id":"2605.29380","date":"2026-05-28","proceeding":"ICML","authors":["Hesam Asadollahzadeh","Feng Liu","Christopher Leckie","Sarah M. Erfani"],"abstract":"Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate **TRACER** (**T**rajectory-**R**obust **A**nchoring for **C**ontrastive **E**ncoder **R**egularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at [https://github.com/HesamAsad/TRACER](https://github.com/HesamAsad/TRACER).","url_abs":"https://arxiv.org/abs/2605.29380","url_pdf":"https://arxiv.org/pdf/2605.29380","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.29380","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.29380"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/HesamAsad/TRACER","reach":null}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"6e97baf23bf18650","entry":"compute_calibration","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"src/visualize.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/src/visualize.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e97baf23bf18650"}},{"code_sha256_prefix":"b2de1386b74212a2","entry":"compute_ldreg_loss","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"src/models/tracer_ldreg.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/src/models/tracer_ldreg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2de1386b74212a2"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"clip/tokenizer.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"5123a768ddd9afa5","entry":"lid_mom_est","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"src/models/tracer_ldreg.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/src/models/tracer_ldreg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5123a768ddd9afa5"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"clip/tokenizer.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/clip/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"b64f67bec6a093b9","entry":"build_model","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"clip/model.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b64f67bec6a093b9"}},{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"clip/loss.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/clip/loss.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"d558de6dd71f4220","entry":"reliability_diagram","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"src/visualize.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/src/visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d558de6dd71f4220"}},{"code_sha256_prefix":"435288e8d2fd445a","entry":"reliability_diagrams","repo":"HesamAsad/TRACER","repo_kind":"found_in_text","path":"src/visualize.py","file_url":"https://github.com/HesamAsad/TRACER/blob/HEAD/src/visualize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"435288e8d2fd445a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}