{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/jatmo-prompt-injection-defense-by-task","title":"Jatmo: Prompt Injection Defense by Task-Specific Finetuning","arxiv_id":"2312.17673","date":"2023-12-29","proceeding":null,"authors":["Julien Piet","Maha Alrashed","Chawin Sitawarin","Sizhe Chen","Zeming Wei","Elizabeth Sun","Basel Alomair","David Wagner"],"abstract":"Large Language Models (LLMs) are attracting significant research attention due to their instruction-following abilities, allowing users and developers to leverage LLMs for a variety of tasks. However, LLMs are vulnerable to prompt-injection attacks: a class of attacks that hijack the model's instruction-following abilities, changing responses to prompts to undesired, possibly malicious ones. In this work, we introduce Jatmo, a method for generating task-specific models resilient to prompt-injection attacks. Jatmo leverages the fact that LLMs can only follow instructions once they have undergone instruction tuning. It harnesses a teacher instruction-tuned model to generate a task-specific dataset, which is then used to fine-tune a base model (i.e., a non-instruction-tuned model). Jatmo only needs a task prompt and a dataset of inputs for the task: it uses the teacher model to generate outputs. For situations with no pre-existing datasets, Jatmo can use a single example, or in some cases none at all, to produce a fully synthetic dataset. Our experiments on seven tasks show that Jatmo models provide similar quality of outputs on their specific task as standard LLMs, while being resilient to prompt injections. The best attacks succeeded in less than 0.5% of cases against our models, versus 87% success rate against GPT-3.5-Turbo. We release Jatmo at https://github.com/wagner-group/prompt-injection-defense.","url_abs":"https://arxiv.org/abs/2312.17673v2","url_pdf":"https://arxiv.org/pdf/2312.17673v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"jatmo-prompt-injection-defense-by-task","repo_url":"https://github.com/wagner-group/prompt-injection-defense","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2312.17673","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.17673"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wagner-group/prompt-injection-defense","reach":{"status":"ok"}}],"summary":{"ran":10,"unverified":8},"by_repo_kind":{"official":{"samples":18,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"58f375594c3a65c3","entry":"call_openai","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/server.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58f375594c3a65c3"}},{"code_sha256_prefix":"afdfabcae731b1b3","entry":"format_finetune_data","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/automatic_pipeline/utils.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/automatic_pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afdfabcae731b1b3"}},{"code_sha256_prefix":"011dc1d185aadcee","entry":"gather_inputs","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/example_tasks/auto_tasks/code_summarization.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/example_tasks/auto_tasks/code_summarization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"011dc1d185aadcee"}},{"code_sha256_prefix":"d8dd0fab84fbf910","entry":"get_formatting_input","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/automatic_pipeline/utils.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/automatic_pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8dd0fab84fbf910"}},{"code_sha256_prefix":"e0da5c3f437fc507","entry":"get_generation_prompt","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/automatic_pipeline/utils.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/automatic_pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0da5c3f437fc507"}},{"code_sha256_prefix":"57b9140b8e2e42bb","entry":"init_servers","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/server.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57b9140b8e2e42bb"}},{"code_sha256_prefix":"af2d33fbd2f2574e","entry":"load_hackaprompt_injections","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/semi_automatic_pipeline/utils.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/semi_automatic_pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af2d33fbd2f2574e"}},{"code_sha256_prefix":"de6f63f3c4a1f298","entry":"normalize_book","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/example_tasks/auto_tasks/translation.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/example_tasks/auto_tasks/translation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de6f63f3c4a1f298"}},{"code_sha256_prefix":"cf548b8291737d65","entry":"perturb_passage","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/semi_automatic_pipeline/utils.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/semi_automatic_pipeline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf548b8291737d65"}},{"code_sha256_prefix":"09ca1d2547f6362b","entry":"write_prompt","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/example_tasks/auto_tasks/review_summarization.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/example_tasks/auto_tasks/review_summarization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09ca1d2547f6362b"}},{"code_sha256_prefix":"f64affb6add891aa","entry":"compare_to_ft_model","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/automatic_pipeline/eval_model.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/automatic_pipeline/eval_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f64affb6add891aa"}},{"code_sha256_prefix":"4877041f759749d1","entry":"format_inputs","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/automatic_pipeline/input_generation.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/automatic_pipeline/input_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4877041f759749d1"}},{"code_sha256_prefix":"aa55ccc6d4abdf48","entry":"gather_inputs","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/example_tasks/auto_tasks/news_summarization.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/example_tasks/auto_tasks/news_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa55ccc6d4abdf48"}},{"code_sha256_prefix":"da6bcc46ae271b58","entry":"gather_inputs","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/example_tasks/auto_tasks/review_summarization.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/example_tasks/auto_tasks/review_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da6bcc46ae271b58"}},{"code_sha256_prefix":"64875c7d8e21226c","entry":"gather_inputs","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/example_tasks/auto_tasks/translation.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/example_tasks/auto_tasks/translation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64875c7d8e21226c"}},{"code_sha256_prefix":"ecbd4b9dae46b569","entry":"get_input_list","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/automatic_pipeline/input_generation.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/automatic_pipeline/input_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecbd4b9dae46b569"}},{"code_sha256_prefix":"45f097951892870c","entry":"load_reviews","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/example_tasks/auto_tasks/review_summarization.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/example_tasks/auto_tasks/review_summarization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45f097951892870c"}},{"code_sha256_prefix":"3b5261f8ca59d546","entry":"rate_completions","repo":"wagner-group/prompt-injection-defense","repo_kind":"official","path":"src/jatmo/server.py","file_url":"https://github.com/wagner-group/prompt-injection-defense/blob/HEAD/src/jatmo/server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b5261f8ca59d546"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}