{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adapting-visual-language-models-for","title":"Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images","arxiv_id":"2403.12570","date":"2024-03-19","proceeding":"CVPR 2024 1","authors":["Chaoqin Huang","Aofan Jiang","Jinghao Feng","Ya zhang","Xinchao Wang","Yanfeng Wang"],"abstract":"Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains. However, the substantial domain divergence between natural and medical images limits the effectiveness of these methodologies in medical anomaly detection. This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, 2.03% and 2.37% for anomaly segmentation, under the zero-shot and few-shot settings, respectively. Source code is available at: https://github.com/MediaBrain-SJTU/MVFA-AD","url_abs":"https://arxiv.org/abs/2403.12570v1","url_pdf":"https://arxiv.org/pdf/2403.12570v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adapting-visual-language-models-for","repo_url":"https://github.com/mediabrain-sjtu/mvfa-ad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"anomaly-classification","task_name":"Anomaly Classification"},{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"anomaly-segmentation","task_name":"Anomaly Segmentation"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.12570","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.12570"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mediabrain-sjtu/mvfa-ad","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/MediaBrain-SJTU/MVFA-AD","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_draft_wrong":4,"ran_violates":1,"ran_honours":1,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b40db86d0bab4263","entry":"ClipAdapter","repo":"MediaBrain-SJTU/MVFA-AD","repo_kind":"official","path":"CLIP/adapter.py","file_url":"https://github.com/MediaBrain-SJTU/MVFA-AD/blob/HEAD/CLIP/adapter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b40db86d0bab4263"}},{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/tokenizer.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"f19962ebb134b3d7","entry":"convert_to_custom_text_state_dict","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/model.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f19962ebb134b3d7"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/model.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/tokenizer.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"f0bead498d7b3068","entry":"get_rot_mat","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"utils.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0bead498d7b3068"}},{"code_sha256_prefix":"4b3a4d17e1df3140","entry":"load_state_dict","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/clip.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/clip.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b3a4d17e1df3140"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/tokenizer.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"09c0abc70500a134","entry":"build_model_from_openai_state_dict","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/model.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09c0abc70500a134"}},{"code_sha256_prefix":"80d50221b83e3ef4","entry":"cos_sim","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"utils.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80d50221b83e3ef4"}},{"code_sha256_prefix":"00c4c440750b8eec","entry":"freeze_batch_norm_2d","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/modified_resnet.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/modified_resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00c4c440750b8eec"}},{"code_sha256_prefix":"5c7cf820e99443ff","entry":"get_model_config","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/clip.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c7cf820e99443ff"}},{"code_sha256_prefix":"d53108edea9d2b46","entry":"load_checkpoint","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/clip.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d53108edea9d2b46"}},{"code_sha256_prefix":"6407d54f96f9b83e","entry":"load_openai_model","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"CLIP/openai.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/CLIP/openai.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6407d54f96f9b83e"}},{"code_sha256_prefix":"9e816aad57db813b","entry":"test","repo":"mediabrain-sjtu/mvfa-ad","repo_kind":"official","path":"train_zero.py","file_url":"https://github.com/mediabrain-sjtu/mvfa-ad/blob/HEAD/train_zero.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e816aad57db813b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}