{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/safeguarding-vision-language-models","title":"Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks","arxiv_id":"2504.01308","date":"2025-04-02","proceeding":null,"authors":["Jiawei Wang","Yushen Zuo","Yuanjun Chai","Zhendong Liu","Yichen Fu","Yichun Feng","Kin-Man Lam"],"abstract":"Vision-Language Models (VLMs) extend the capabilities of Large Language Models (LLMs) by incorporating visual information, yet they remain vulnerable to jailbreak attacks, especially when processing noisy or corrupted images. Although existing VLMs adopt security measures during training to mitigate such attacks, vulnerabilities associated with noise-augmented visual inputs are overlooked. In this work, we identify that missing noise-augmented training causes critical security gaps: many VLMs are susceptible to even simple perturbations such as Gaussian noise. To address this challenge, we propose Robust-VLGuard, a multimodal safety dataset with aligned / misaligned image-text pairs, combined with noise-augmented fine-tuning that reduces attack success rates while preserving functionality of VLM. For stronger optimization-based visual perturbation attacks, we propose DiffPure-VLM, leveraging diffusion models to convert adversarial perturbations into Gaussian-like noise, which can be defended by VLMs with noise-augmented safety fine-tuning. Experimental results demonstrate that the distribution-shifting property of diffusion model aligns well with our fine-tuned VLMs, significantly mitigating adversarial perturbations across varying intensities. The dataset and code are available at https://github.com/JarvisUSTC/DiffPure-RobustVLM.","url_abs":"https://arxiv.org/abs/2504.01308v1","url_pdf":"https://arxiv.org/pdf/2504.01308v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"safeguarding-vision-language-models","repo_url":"https://github.com/jarvisustc/diffpure-robustvlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"adopt","method_name":"ADOPT"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.01308","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.01308"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/JarvisUSTC/DiffPure-RobustVLM","reach":null}],"summary":{"ran":1,"ran_honours":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"7cb309e61263531a","entry":"PurificationForward","repo":"JarvisUSTC/DiffPure-RobustVLM","repo_kind":"official","path":"purification.py","file_url":"https://github.com/JarvisUSTC/DiffPure-RobustVLM/blob/HEAD/purification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7cb309e61263531a"}},{"code_sha256_prefix":"0601c260f2ed6579","entry":"clf2diff","repo":"JarvisUSTC/DiffPure-RobustVLM","repo_kind":"official","path":"purification.py","file_url":"https://github.com/JarvisUSTC/DiffPure-RobustVLM/blob/HEAD/purification.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0601c260f2ed6579"}},{"code_sha256_prefix":"4c0585298faf5ef1","entry":"diff2clf","repo":"JarvisUSTC/DiffPure-RobustVLM","repo_kind":"official","path":"purification.py","file_url":"https://github.com/JarvisUSTC/DiffPure-RobustVLM/blob/HEAD/purification.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c0585298faf5ef1"}},{"code_sha256_prefix":"b0c47ff6d5f12b4f","entry":"get_beta_schedule","repo":"JarvisUSTC/DiffPure-RobustVLM","repo_kind":"official","path":"purification.py","file_url":"https://github.com/JarvisUSTC/DiffPure-RobustVLM/blob/HEAD/purification.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0c47ff6d5f12b4f"}},{"code_sha256_prefix":"3eec6934f3697dde","entry":"normalize","repo":"JarvisUSTC/DiffPure-RobustVLM","repo_kind":"official","path":"purification.py","file_url":"https://github.com/JarvisUSTC/DiffPure-RobustVLM/blob/HEAD/purification.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3eec6934f3697dde"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}