{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-jailbreak-tax-how-useful-are-your","title":"The Jailbreak Tax: How Useful are Your Jailbreak Outputs?","arxiv_id":"2504.10694","date":"2025-04-14","proceeding":null,"authors":["Kristina Nikolić","Luze Sun","Jie Zhang","Florian Tramèr"],"abstract":"Jailbreak attacks bypass the guardrails of large language models to produce harmful outputs. In this paper, we ask whether the model outputs produced by existing jailbreaks are actually useful. For example, when jailbreaking a model to give instructions for building a bomb, does the jailbreak yield good instructions? Since the utility of most unsafe answers (e.g., bomb instructions) is hard to evaluate rigorously, we build new jailbreak evaluation sets with known ground truth answers, by aligning models to refuse questions related to benign and easy-to-evaluate topics (e.g., biology or math). Our evaluation of eight representative jailbreaks across five utility benchmarks reveals a consistent drop in model utility in jailbroken responses, which we term the jailbreak tax. For example, while all jailbreaks we tested bypass guardrails in models aligned to refuse to answer math, this comes at the expense of a drop of up to 92% in accuracy. Overall, our work proposes the jailbreak tax as a new important metric in AI safety, and introduces benchmarks to evaluate existing and future jailbreaks. We make the benchmark available at https://github.com/ethz-spylab/jailbreak-tax","url_abs":"https://arxiv.org/abs/2504.10694v1","url_pdf":"https://arxiv.org/pdf/2504.10694v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-jailbreak-tax-how-useful-are-your","repo_url":"https://github.com/ethz-spylab/jailbreak-tax","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"math","task_name":"Math"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.10694","atlas_url":"https://app.syntology.ai/?focus=2504.10694","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.10694"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ethz-spylab/jailbreak-tax","reach":null}],"summary":{"ran_draft_wrong":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"6ea4368f35601b99","entry":"custom_jailbreak","repo":"ethz-spylab/jailbreak-tax","repo_kind":"official","path":"jb_tax_example_file.py","file_url":"https://github.com/ethz-spylab/jailbreak-tax/blob/HEAD/jb_tax_example_file.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ea4368f35601b99"}},{"code_sha256_prefix":"0040c5f48995055a","entry":"extract_response_text","repo":"ethz-spylab/jailbreak-tax","repo_kind":"official","path":"Utils/jb_tax_grading_utils.py","file_url":"https://github.com/ethz-spylab/jailbreak-tax/blob/HEAD/Utils/jb_tax_grading_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0040c5f48995055a"}},{"code_sha256_prefix":"44d9f020d142c103","entry":"get_output_directory","repo":"ethz-spylab/jailbreak-tax","repo_kind":"official","path":"run_jb_tax.py","file_url":"https://github.com/ethz-spylab/jailbreak-tax/blob/HEAD/run_jb_tax.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44d9f020d142c103"}},{"code_sha256_prefix":"5c31b8c6ab28be93","entry":"get_question_status","repo":"ethz-spylab/jailbreak-tax","repo_kind":"official","path":"Utils/jb_tax_grading_utils.py","file_url":"https://github.com/ethz-spylab/jailbreak-tax/blob/HEAD/Utils/jb_tax_grading_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c31b8c6ab28be93"}},{"code_sha256_prefix":"fe8573702ddd23b1","entry":"setup_stage_directory","repo":"ethz-spylab/jailbreak-tax","repo_kind":"official","path":"run_jb_tax.py","file_url":"https://github.com/ethz-spylab/jailbreak-tax/blob/HEAD/run_jb_tax.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe8573702ddd23b1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}