{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-llms-separate-instructions-from-data-and","title":"Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?","arxiv_id":"2403.06833","date":"2024-03-11","proceeding":null,"authors":["Egor Zverev","Sahar Abdelnabi","Soroush Tabesh","Mario Fritz","Christoph H. Lampert"],"abstract":"Instruction-tuned Large Language Models (LLMs) show impressive results in numerous practical applications, but they lack essential safety features that are common in other areas of computer science, particularly an explicit separation of instructions and data. This makes them vulnerable to manipulations such as indirect prompt injections and generally unsuitable for safety-critical tasks. Surprisingly, there is currently no established definition or benchmark to quantify this phenomenon. In this work, we close this gap by introducing a formal measure for instruction-data separation and an empirical variant that is calculable from a model's outputs. We also present a new dataset, SEP, that allows estimating the measure for real-world models. Our results on various LLMs show that the problem of instruction-data separation is real: all models fail to achieve high separation, and canonical mitigation techniques, such as prompt engineering and fine-tuning, either fail to substantially improve separation or reduce model utility. The source code and SEP dataset are openly accessible at https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed.","url_abs":"https://arxiv.org/abs/2403.06833v3","url_pdf":"https://arxiv.org/pdf/2403.06833v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-llms-separate-instructions-from-data-and","repo_url":"https://github.com/egozverev/shold-it-be-executed-or-processed","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"can-llms-separate-instructions-from-data-and","repo_url":"https://github.com/facebookresearch/meta_secalign","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.06833","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.06833"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/egozverev/shold-it-be-executed-or-processed","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/meta_secalign","reach":null}],"summary":{"ran":12},"by_repo_kind":{"official":{"samples":12,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"92f7f215ffba3e50","entry":"assemble_probe_dataset","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"SEP_dataset/insert_probes.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/SEP_dataset/insert_probes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92f7f215ffba3e50"}},{"code_sha256_prefix":"14b5cdb7d1ca71c0","entry":"flatten_dataset","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"SEP_dataset/insert_probes.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/SEP_dataset/insert_probes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14b5cdb7d1ca71c0"}},{"code_sha256_prefix":"ed71b768b354fced","entry":"format_prompt","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"model_eval/get_model_outputs.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/model_eval/get_model_outputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed71b768b354fced"}},{"code_sha256_prefix":"7dc35b0ed37e0372","entry":"load_config","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"model_eval/get_model_outputs.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/model_eval/get_model_outputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7dc35b0ed37e0372"}},{"code_sha256_prefix":"0d9367e82ace5842","entry":"load_config","repo":"egozverev/shold-it-be-executed-or-processed","repo_kind":"official","path":"SEP_dataset/utils.py","file_url":"https://github.com/egozverev/shold-it-be-executed-or-processed/blob/HEAD/SEP_dataset/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d9367e82ace5842"}},{"code_sha256_prefix":"4858a0fde9a72faa","entry":"load_data","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"model_eval/get_model_outputs.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/model_eval/get_model_outputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4858a0fde9a72faa"}},{"code_sha256_prefix":"c822a8f32726ba36","entry":"load_json_data","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"SEP_dataset/utils.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/SEP_dataset/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c822a8f32726ba36"}},{"code_sha256_prefix":"8937d8bba4e3d5d0","entry":"load_json_files_from_dir","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"model_eval/analyze_results.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/model_eval/analyze_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8937d8bba4e3d5d0"}},{"code_sha256_prefix":"37a9ce439024e252","entry":"process_gen_output","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"openai_utils.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/openai_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37a9ce439024e252"}},{"code_sha256_prefix":"e08792f2a461b463","entry":"read_file","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"SEP_dataset/utils.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/SEP_dataset/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e08792f2a461b463"}},{"code_sha256_prefix":"a4770293aaa8cd1b","entry":"retry_with_exponential_backoff","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"openai_utils.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/openai_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4770293aaa8cd1b"}},{"code_sha256_prefix":"b759ac5d6c361342","entry":"try_processing_json_str","repo":"egozverev/Shold-It-Be-Executed-Or-Processed","repo_kind":"official","path":"openai_utils.py","file_url":"https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed/blob/HEAD/openai_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b759ac5d6c361342"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}