Papers › Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations

Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations

24 Mar 2025CVPR 2025 1arXiv:2503.18817archive 2025-07-28

JEONGHYEON KIM, Sangheum Hwang

Prior research on out-of-distribution detection (OoDD) has primarily focused on single-modality models. Recently, with the advent of large-scale pretrained vision-language models such as CLIP, OoDD methods utilizing such multi-modal representations through zero-shot and prompt learning strategies have emerged. However, these methods typically involve either freezing the pretrained weights or only partially tuning them, which can be suboptimal for downstream datasets. In this paper, we highlight that multi-modal fine-tuning (MMFT) can achieve notable OoDD performance. Despite some recent works demonstrating the impact of fine-tuning methods for OoDD, there remains significant potential for performance improvement. We investigate the limitation of naïve fine-tuning methods, examining why they fail to fully leverage the pretrained knowledge. Our empirical analysis suggests that this issue could stem from the modality gap within in-distribution (ID) embeddings. To address this, we propose a training objective that enhances cross-modal alignment by regularizing the distances between image and text embeddings of ID data. This adjustment helps in better utilizing pretrained textual information by aligning similar semantics from different modalities (i.e., text and image) more closely in the hyperspherical representation space. We theoretically demonstrate that the proposed regularization corresponds to the maximum likelihood estimation of an energy-based model on a hypersphere. Utilizing ImageNet-1k OoD benchmark datasets, we show that our method, combined with post-hoc OoDD approaches leveraging pretrained knowledge (e.g., NegLabel), significantly outperforms existing methods, achieving state-of-the-art OoDD performance and leading ID accuracy.

PaperPDFConference PDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

ma-kjh/CMA-OoDD officialmentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space (arXiv:2105.01879; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified) and Negative Label Guided OOD Detection with Pretrained Vision-Language Models (arXiv:2403.20078; that paper's own run record: 9 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image ClassificationOut-of-Distribution DetectionPrompt Learningcross-modal alignment

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Image Classification ImageNet CMA(ViT-B/16) Top 1 Accuracy 82.64% #517 of 1060 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs Places CMA(ViT-B/16, NegLabel) AUROC 93.11 #1 of 25 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs Places CMA(ViT-B/16, NegLabel) FPR95 27.65 #1 of 25 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs SUN CMA(ViT-B/16, NegLabel) AUROC 96.36 #1 of 22 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs SUN CMA(ViT-B/16, NegLabel) FPR95 16.84 #1 of 22 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs Textures CMA(ViT-B/16, NegLabel) AUROC 91.64 #16 of 34 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs Textures CMA(ViT-B/16, NegLabel) FPR95 33.58 #16 of 34 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs iNaturalist CMA(ViT-B/16, NegLabel) AUROC 99.62 #2 of 28 Archive leaderboard report
Out-of-Distribution Detection ImageNet-1k vs iNaturalist CMA(ViT-B/16, NegLabel) FPR95 1.65 #2 of 28 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

CLIP

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections