Papers › Context-Aware Robust Fine-Tuning
Context-Aware Robust Fine-Tuning
Xiaofeng Mao, Yuefeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, Zhao Li
Contrastive Language-Image Pre-trained (CLIP) models have zero-shot ability of classifying an image belonging to "[CLASS]" by using similarity between the image and the prompt sentence "a [CONTEXT] of [CLASS]". Based on exhaustive text cues in "[CONTEXT]", CLIP model is aware of different contexts, e.g. background, style, viewpoint, and exhibits unprecedented robustness against a wide range of distribution shifts. However, recent works find further fine-tuning of CLIP models improves accuracy but sacrifices the robustness on downstream tasks. We conduct an empirical investigation to show fine-tuning will corrupt the context-aware ability of pre-trained CLIP features. To solve this problem, we propose Context-Aware Robust Fine-tuning (CAR-FT). CAR-FT regularizes the model during fine-tuning to capture the context information. Specifically, we use zero-shot prompt weights to get the context distribution contained in the image. By minimizing the Kullback-Leibler Divergence (KLD) between context distributions induced by original/fine-tuned CLIP models, CAR-FT makes the context-aware ability of CLIP inherited into downstream tasks, and achieves both higher In-Distribution (ID) and Out-Of-Distribution (OOD) accuracy. The experimental results show CAR-FT achieves superior robustness on five OOD test datasets of ImageNet, and meanwhile brings accuracy gains on nine downstream tasks. Additionally, CAR-FT surpasses previous Domain Generalization (DG) methods and gets 78.5% averaged accuracy on DomainBed benchmark, building the new state-of-the-art.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Domain Generalization | DomainNet | CAR-FT (CLIP, ViT-B/16) | Average Accuracy | 62.5 | #6 of 38 | Archive leaderboard | report |
| Domain Generalization | ImageNet-A | CAR-FT (CLIP, ViT-L/14@336px) | Top-1 accuracy % | 81.5 | #4 of 39 | Archive leaderboard | report |
| Domain Generalization | ImageNet-R | CAR-FT (CLIP, ViT-L/14@336px) | Top-1 Error Rate | 10.3 | #3 of 39 | Archive leaderboard | report |
| Domain Generalization | ImageNet-Sketch | CAR-FT (CLIP, ViT-L/14@336px) | Top-1 accuracy | 65.5 | #3 of 20 | Archive leaderboard | report |
| Domain Generalization | Office-Home | CAR-FT (CLIP, ViT-B/16) | Average Accuracy | 85.7 | #6 of 45 | Archive leaderboard | report |
| Domain Generalization | PACS | CAR-FT (CLIP, ViT-B/16) | Average Accuracy | 96.8 | #10 of 133 | Archive leaderboard | report |
| Domain Generalization | TerraIncognita | CAR-FT (CLIP, ViT-B/16) | Average Accuracy | 61.9 | #4 of 30 | Archive leaderboard | report |
| Domain Generalization | VLCS | CAR-FT (CLIP, ViT-B/16) | Average Accuracy | 85.5 | #1 of 37 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections