{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ignore-this-title-and-hackaprompt-exposing","title":"Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition","arxiv_id":"2311.16119","date":"2023-10-24","proceeding":null,"authors":["Sander Schulhoff","Jeremy Pinto","Anaum Khan","Louis-François Bouchard","Chenglei Si","Svetlina Anati","Valen Tagliabue","Anson Liu Kost","Christopher Carnahan","Jordan Boyd-Graber"],"abstract":"Large Language Models (LLMs) are deployed in interactive contexts with direct user engagement, such as chatbots and writing assistants. These deployments are vulnerable to prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive taxonomical ontology of the types of adversarial prompts.","url_abs":"https://arxiv.org/abs/2311.16119v3","url_pdf":"https://arxiv.org/pdf/2311.16119v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ignore-this-title-and-hackaprompt-exposing","repo_url":"https://github.com/trigaten/learn_prompting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"ignore-this-title-and-hackaprompt-exposing","repo_url":"https://github.com/promptlabs/hackaprompt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"ignore-this-title-and-hackaprompt-exposing","repo_url":"https://github.com/lostoxygen/llm-confidentiality","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"ontology","method_name":"Ontology"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.16119","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.16119"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lostoxygen/llm-confidentiality","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/promptlabs/hackaprompt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/trigaten/learn_prompting","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":2,"unverified":6},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f2436d35bad4ef13","entry":"format_evaluation","repo":"promptlabs/hackaprompt","repo_kind":"official","path":"hackaprompt/gradio_app.py","file_url":"https://github.com/promptlabs/hackaprompt/blob/HEAD/hackaprompt/gradio_app.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2436d35bad4ef13"}},{"code_sha256_prefix":"3ca22df3ad4f2a89","entry":"save_response","repo":"promptlabs/hackaprompt","repo_kind":"official","path":"hackaprompt/gradio_app.py","file_url":"https://github.com/promptlabs/hackaprompt/blob/HEAD/hackaprompt/gradio_app.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ca22df3ad4f2a89"}},{"code_sha256_prefix":"56d6fdeaf88aced6","entry":"identity_prompt","repo":"lostoxygen/llm-confidentiality","repo_kind":"listed","path":"framework/defenses.py","file_url":"https://github.com/lostoxygen/llm-confidentiality/blob/HEAD/framework/defenses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56d6fdeaf88aced6"}},{"code_sha256_prefix":"57f17a1aa7ba1c17","entry":"match_attack","repo":"lostoxygen/llm-confidentiality","repo_kind":"listed","path":"framework/attacks.py","file_url":"https://github.com/lostoxygen/llm-confidentiality/blob/HEAD/framework/attacks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57f17a1aa7ba1c17"}},{"code_sha256_prefix":"3ee5e8df5cd145e2","entry":"match_defense","repo":"lostoxygen/llm-confidentiality","repo_kind":"listed","path":"framework/defenses.py","file_url":"https://github.com/lostoxygen/llm-confidentiality/blob/HEAD/framework/defenses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ee5e8df5cd145e2"}},{"code_sha256_prefix":"34b2f55d744ab028","entry":"obfuscation","repo":"lostoxygen/llm-confidentiality","repo_kind":"listed","path":"framework/attacks.py","file_url":"https://github.com/lostoxygen/llm-confidentiality/blob/HEAD/framework/attacks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34b2f55d744ab028"}},{"code_sha256_prefix":"33d66352103672e0","entry":"payload_splitting","repo":"lostoxygen/llm-confidentiality","repo_kind":"listed","path":"framework/attacks.py","file_url":"https://github.com/lostoxygen/llm-confidentiality/blob/HEAD/framework/attacks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33d66352103672e0"}},{"code_sha256_prefix":"3daf4627c3dd3630","entry":"seq_enclosure","repo":"lostoxygen/llm-confidentiality","repo_kind":"listed","path":"framework/defenses.py","file_url":"https://github.com/lostoxygen/llm-confidentiality/blob/HEAD/framework/defenses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3daf4627c3dd3630"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}