{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-11817","title":"Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code","arxiv_id":"2606.11817","date":"2026-06-10","proceeding":null,"authors":["Yitong Zhang","Shiteng Lu","Jia Li"],"abstract":"Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.","url_abs":"https://arxiv.org/abs/2606.11817","url_pdf":"https://arxiv.org/pdf/2606.11817","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2606.11817","atlas_url":"https://app.syntology.ai/?focus=2606.11817","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.11817"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield","reach":null}],"summary":{"ran":13,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b2911c9f56f3f47c","entry":"build_mr_user_prompt","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/evaluate_mr.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/evaluate_mr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2911c9f56f3f47c"}},{"code_sha256_prefix":"4bf38ef0ebadef49","entry":"build_output_root","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/merge_shards.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/merge_shards.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bf38ef0ebadef49"}},{"code_sha256_prefix":"748e861da66aaa33","entry":"build_result_file_name","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/run_main.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/run_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"748e861da66aaa33"}},{"code_sha256_prefix":"a4b7dce535911857","entry":"choose_shards","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/merge_shards.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/merge_shards.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4b7dce535911857"}},{"code_sha256_prefix":"4c4af1cda9f9e155","entry":"get_result_prefix","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/run_main.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/run_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c4af1cda9f9e155"}},{"code_sha256_prefix":"cbdd6c14a82c9ee1","entry":"has_malicious_content","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Defense/build_training_data.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Defense/build_training_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbdd6c14a82c9ee1"}},{"code_sha256_prefix":"aecb899a3b7dc24b","entry":"infer_resume_index","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/run_main.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/run_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aecb899a3b7dc24b"}},{"code_sha256_prefix":"5285b0b1a83df036","entry":"is_python","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Defense/build_training_data.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Defense/build_training_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5285b0b1a83df036"}},{"code_sha256_prefix":"719c374000448a22","entry":"list_shard_items","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/merge_shards.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/merge_shards.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"719c374000448a22"}},{"code_sha256_prefix":"4d2567d9174fe3cb","entry":"load_malicious_keywords","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Defense/build_training_data.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Defense/build_training_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d2567d9174fe3cb"}},{"code_sha256_prefix":"e3598a0158070938","entry":"strip_forced_prefix","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/evaluate_asr.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/evaluate_asr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3598a0158070938"}},{"code_sha256_prefix":"d74ef718b83da46a","entry":"truncate_text","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/evaluate_asr.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/evaluate_asr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d74ef718b83da46a"}},{"code_sha256_prefix":"fd2cfb4a6f4f355b","entry":"truncate_text","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/evaluate_mr.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/evaluate_mr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd2cfb4a6f4f355b"}},{"code_sha256_prefix":"d5c2e4e4ec38a32f","entry":"get_evaluation","repo":"TsinghuaISE/CodeSpear-CodeShield","repo_kind":"found_in_text","path":"Attack/LocalModels/script/evaluate_asr.py","file_url":"https://github.com/TsinghuaISE/CodeSpear-CodeShield/blob/HEAD/Attack/LocalModels/script/evaluate_asr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5c2e4e4ec38a32f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}