{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/safety-mirage-how-spurious-correlations","title":"Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-tuning","arxiv_id":"2503.11832","date":"2025-03-14","proceeding":null,"authors":["YiWei Chen","Yuguang Yao","Yihua Zhang","Bingquan Shen","Gaowen Liu","Sijia Liu"],"abstract":"Recent vision-language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images. However, their susceptibility to generating harmful content when exposed to unsafe queries raises critical safety concerns. While current alignment strategies primarily rely on supervised safety fine-tuning with curated datasets, we identify a fundamental limitation we call the \"safety mirage\" where supervised fine-tuning inadvertently reinforces spurious correlations between superficial textual patterns and safety responses, rather than fostering deep, intrinsic mitigation of harm. We show that these spurious correlations leave fine-tuned VLMs vulnerable even to a simple one-word modification-based attack, where substituting a single word in text queries with a spurious correlation-inducing alternative can effectively bypass safeguards. Additionally, these correlations contribute to the over prudence, causing fine-tuned VLMs to refuse benign queries unnecessarily. To address this issue, we show machine unlearning (MU) as a powerful alternative to supervised safety fine-tuning as it avoids biased feature-label mappings and directly removes harmful knowledge from VLMs while preserving their general capabilities. Extensive evaluations across safety benchmarks show that under one-word attacks, MU-based alignment reduces the attack success rate by up to 60.17% and cuts unnecessary rejections by over 84.20%. Codes are available at https://github.com/OPTML-Group/VLM-Safety-MU. WARNING: There exist AI generations that may be offensive in nature.","url_abs":"https://arxiv.org/abs/2503.11832v1","url_pdf":"https://arxiv.org/pdf/2503.11832v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"safety-mirage-how-spurious-correlations","repo_url":"https://github.com/optml-group/vlm-safety-mu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-unlearning","task_name":"Machine Unlearning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.11832","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.11832"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/optml-group/vlm-safety-mu","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":2,"ran_violates":1,"ran_honours":1,"ran_fixture":2,"unverified":5},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"found_in_text":{"samples":10,"ran":5,"repositories":1},"community":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/eval/model_vqa_loader.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/eval/model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"optml-group/vlm-safety-mu","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/optml-group/vlm-safety-mu/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/eval/model_vqa.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/eval/model_vqa_mmbench.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"188b8fe064496487","entry":"lr_schedule","repo":"alewarne/MachineUnlearning","repo_kind":"community","path":"Unlearner/DNNUnlearner.py","file_url":"https://github.com/alewarne/MachineUnlearning/blob/HEAD/Unlearner/DNNUnlearner.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"188b8fe064496487"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"optml-group/vlm-safety-mu","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/optml-group/vlm-safety-mu/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"optml-group/vlm-safety-mu","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/optml-group/vlm-safety-mu/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/eval/model_vqa.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/train/llava_unlearn_full_trainer.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/train/llava_unlearn_full_trainer.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/train/llava_unlearn_full_trainer.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/train/llava_unlearn_full_trainer.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"fa1225dfac92bc0d","entry":"get_peft_state_maybe_zero_3","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/train/train_unlearn.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/train/train_unlearn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa1225dfac92bc0d"}},{"code_sha256_prefix":"1c53657305b66e9f","entry":"get_peft_state_non_lora_maybe_zero_3","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/train/train_unlearn.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/train/train_unlearn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c53657305b66e9f"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/train/llava_unlearn_full_trainer.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/train/llava_unlearn_full_trainer.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"OPTML-Group/VLM-Safety-Unlearn","repo_kind":"found_in_text","path":"llava/train/train_unlearn.py","file_url":"https://github.com/OPTML-Group/VLM-Safety-Unlearn/blob/HEAD/llava/train/train_unlearn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}