{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/steering-away-from-harm-an-adaptive-approach","title":"Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks","arxiv_id":"2411.16721","date":"2024-11-23","proceeding":"CVPR 2025 1","authors":["Han Wang","Gang Wang","huan zhang"],"abstract":"Vision Language Models (VLMs) can produce unintended and harmful content when exposed to adversarial attacks, particularly because their vision capabilities create new vulnerabilities. Existing defenses, such as input preprocessing, adversarial training, and response evaluation-based methods, are often impractical for real-world deployment due to their high costs. To address this challenge, we propose ASTRA, an efficient and effective defense by adaptively steering models away from adversarial feature directions to resist VLM attacks. Our key procedures involve finding transferable steering vectors representing the direction of harmful response and applying adaptive activation steering to remove these directions at inference time. To create effective steering vectors, we randomly ablate the visual tokens from the adversarial images and identify those most strongly associated with jailbreaks. These tokens are then used to construct steering vectors. During inference, we perform the adaptive steering method that involves the projection between the steering vectors and calibrated activation, resulting in little performance drops on benign inputs while strongly avoiding harmful outputs under adversarial inputs. Extensive experiments across multiple models and baselines demonstrate our state-of-the-art performance and high efficiency in mitigating jailbreak risks. Additionally, ASTRA exhibits good transferability, defending against unseen attacks (i.e., structured-based attack, perturbation-based attack with project gradient descent variants, and text-only attack). Our code is available at \\url{https://github.com/ASTRAL-Group/ASTRA}.","url_abs":"https://arxiv.org/abs/2411.16721v3","url_pdf":"https://arxiv.org/pdf/2411.16721v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"steering-away-from-harm-an-adaptive-approach","repo_url":"https://github.com/ASTRAL-Group/ASTRA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.16721","atlas_url":"https://app.syntology.ai/?focus=2411.16721","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.16721"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ASTRAL-Group/ASTRA","reach":null}],"summary":{"ran_draft_wrong":1,"ran_violates":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"2c5fc2384c283cdb","entry":"get_options","repo":"ASTRAL-Group/ASTRA","repo_kind":"official","path":"utility_eval/minigpt_mmbench.py","file_url":"https://github.com/ASTRAL-Group/ASTRA/blob/HEAD/utility_eval/minigpt_mmbench.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c5fc2384c283cdb"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"ASTRAL-Group/ASTRA","repo_kind":"official","path":"utility_eval/minigpt_mmbench.py","file_url":"https://github.com/ASTRAL-Group/ASTRA/blob/HEAD/utility_eval/minigpt_mmbench.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"3aada9ef083faccd","entry":"load_image_from_base64","repo":"ASTRAL-Group/ASTRA","repo_kind":"official","path":"utility_eval/minigpt_mmbench.py","file_url":"https://github.com/ASTRAL-Group/ASTRA/blob/HEAD/utility_eval/minigpt_mmbench.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3aada9ef083faccd"}},{"code_sha256_prefix":"b663ef0730c62263","entry":"load_images_from_folder","repo":"ASTRAL-Group/ASTRA","repo_kind":"official","path":"extract_attr/extract_qwen_jb_attr.py","file_url":"https://github.com/ASTRAL-Group/ASTRA/blob/HEAD/extract_attr/extract_qwen_jb_attr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b663ef0730c62263"}},{"code_sha256_prefix":"225d9b9143eece32","entry":"load_images_from_folder","repo":"ASTRAL-Group/ASTRA","repo_kind":"official","path":"steer_eval/steering_qwen_toxic.py","file_url":"https://github.com/ASTRAL-Group/ASTRA/blob/HEAD/steer_eval/steering_qwen_toxic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"225d9b9143eece32"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}