{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-26096","title":"ALMGuard: Safety Shortcuts and Where to Find Them as Guardrails for Audio-Language Models","arxiv_id":"2510.26096","date":"2025-10-30","proceeding":"NeurIPS","authors":["Weifei Jin","Yuxin Cao","Junjie Su","Minhui Xue","Jie Hao","Ke Xu","Jin Song Dong","Derui Wang"],"abstract":"Recent advances in Audio-Language Models (ALMs) have significantly improved multimodal understanding capabilities. However, the introduction of the audio modality also brings new and unique vulnerability vectors. Previous studies have proposed jailbreak attacks that specifically target ALMs, revealing that defenses directly transferred from traditional audio adversarial attacks or text-based Large Language Model (LLM) jailbreaks are largely ineffective against these ALM-specific threats. To address this issue, we propose ALMGuard, the first defense framework tailored to ALMs. Based on the assumption that safety-aligned shortcuts naturally exist in ALMs, we design a method to identify universal Shortcut Activation Perturbations (SAPs) that serve as triggers that activate the safety shortcuts to safeguard ALMs at inference time. To better sift out effective triggers while preserving the model's utility on benign tasks, we further propose Mel-Gradient Sparse Mask (M-GSM), which restricts perturbations to Mel-frequency bins that are sensitive to jailbreaks but insensitive to speech understanding. Both theoretical analyses and empirical results demonstrate the robustness of our method against both seen and unseen attacks. Overall, \\MethodName reduces the average success rate of advanced ALM-specific jailbreak attacks to 4.6% across four models, while maintaining comparable utility on benign benchmarks, establishing it as the new state of the art. Our code and data are available at https://github.com/WeifeiJin/ALMGuard.","url_abs":"https://arxiv.org/abs/2510.26096","url_pdf":"https://arxiv.org/pdf/2510.26096","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.26096","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.26096"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/WeifeiJin/ALMGuard","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"unverified":11},"by_repo_kind":{"found_in_text":{"samples":12,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"e529c9641c178fe0","entry":"sinusoids","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"whisper/model.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/whisper/model.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e529c9641c178fe0"}},{"code_sha256_prefix":"f0ae8c33b1126961","entry":"extract_prefix_number","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"AIR-bench/cal_score.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/AIR-bench/cal_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0ae8c33b1126961"}},{"code_sha256_prefix":"9f41cd96267d6bc7","entry":"get_input_embeds","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f41cd96267d6bc7"}},{"code_sha256_prefix":"c87203973f499743","entry":"get_sorrybench_pipeline","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"llm_evaluation.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/llm_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c87203973f499743"}},{"code_sha256_prefix":"80abc0b281d75575","entry":"load_audio","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"whisper/audio.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/whisper/audio.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80abc0b281d75575"}},{"code_sha256_prefix":"f8a39516628f22a9","entry":"load_perturb","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"eval_qwen.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/eval_qwen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8a39516628f22a9"}},{"code_sha256_prefix":"73c92993abbf3787","entry":"load_perturb","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73c92993abbf3787"}},{"code_sha256_prefix":"12c6c409a1e359c3","entry":"mel_filters","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"whisper/audio.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/whisper/audio.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12c6c409a1e359c3"}},{"code_sha256_prefix":"14a82fce86cab897","entry":"openai_tts","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"datasets/openai_tts.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/datasets/openai_tts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14a82fce86cab897"}},{"code_sha256_prefix":"9f13de1c23e4003d","entry":"pad_or_trim","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"whisper/audio.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/whisper/audio.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f13de1c23e4003d"}},{"code_sha256_prefix":"c60dd1ae7e5aaea9","entry":"qwen_infer","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"eval_qwen.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/eval_qwen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c60dd1ae7e5aaea9"}},{"code_sha256_prefix":"1dea33e18fd05a0f","entry":"sorrybench_evaluate","repo":"WeifeiJin/ALMGuard","repo_kind":"found_in_text","path":"llm_evaluation.py","file_url":"https://github.com/WeifeiJin/ALMGuard/blob/HEAD/llm_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1dea33e18fd05a0f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.SD","source":"arxiv_api"},"syntology_extracted_results":null}