{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hijacking-large-language-models-via","title":"Hijacking Large Language Models via Adversarial In-Context Learning","arxiv_id":"2311.09948","date":"2023-11-16","proceeding":null,"authors":["Xiangyu Zhou","Yao Qiang","Saleh Zare Zade","Prashant Khanduri","Dongxiao Zhu"],"abstract":"In-context learning (ICL) has emerged as a powerful paradigm leveraging LLMs for specific downstream tasks by utilizing labeled examples as demonstrations (demos) in the preconditioned prompts. Despite its promising performance, crafted adversarial attacks pose a notable threat to the robustness of LLMs. Existing attacks are either easy to detect, require a trigger in user input, or lack specificity towards ICL. To address these issues, this work introduces a novel transferable prompt injection attack against ICL, aiming to hijack LLMs to generate the target output or elicit harmful responses. In our threat model, the hacker acts as a model publisher who leverages a gradient-based prompt search method to learn and append imperceptible adversarial suffixes to the in-context demos via prompt injection. We also propose effective defense strategies using a few shots of clean demos, enhancing the robustness of LLMs during ICL. Extensive experimental results across various classification and jailbreak tasks demonstrate the effectiveness of the proposed attack and defense strategies. This work highlights the significant security vulnerabilities of LLMs during ICL and underscores the need for further in-depth studies.","url_abs":"https://arxiv.org/abs/2311.09948v3","url_pdf":"https://arxiv.org/pdf/2311.09948v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hijacking-large-language-models-via","repo_url":"https://github.com/RookieZxy/GGI-attack","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"specificity","task_name":"Specificity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.09948","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09948"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RookieZxy/GGI-attack","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"2e16b01dcf774c0d","entry":"split_string","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"GGI-attack/llm_attacks/ggi/string_utils.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/GGI-attack/llm_attacks/ggi/string_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e16b01dcf774c0d"}},{"code_sha256_prefix":"7c5149fc31eebb8f","entry":"trimmed_mean","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"Baseline-attack/utils/constants.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/Baseline-attack/utils/constants.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c5149fc31eebb8f"}},{"code_sha256_prefix":"54af014068eb7eff","entry":"tuple_type","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"Baseline-attack/utils/constants.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/Baseline-attack/utils/constants.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54af014068eb7eff"}},{"code_sha256_prefix":"08e65c38a5b1c5a1","entry":"get_embedding_layer","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"GGI-attack/llm_attacks/base/attack_manager.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/GGI-attack/llm_attacks/base/attack_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"08e65c38a5b1c5a1"}},{"code_sha256_prefix":"2703f8b621b73255","entry":"get_embedding_matrix","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"GGI-attack/llm_attacks/base/attack_manager.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/GGI-attack/llm_attacks/base/attack_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2703f8b621b73255"}},{"code_sha256_prefix":"33377834cc7d0c84","entry":"get_embeddings","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"GGI-attack/llm_attacks/base/attack_manager.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/GGI-attack/llm_attacks/base/attack_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33377834cc7d0c84"}},{"code_sha256_prefix":"8de79cc4c0ca62d1","entry":"get_filtered_cands","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"GGI-attack/llm_attacks/ggi/opt_utils.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/GGI-attack/llm_attacks/ggi/opt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8de79cc4c0ca62d1"}},{"code_sha256_prefix":"94d7ddaf0169dbf1","entry":"sample_control","repo":"RookieZxy/GGI-attack","repo_kind":"official","path":"GGI-attack/llm_attacks/ggi/opt_utils.py","file_url":"https://github.com/RookieZxy/GGI-attack/blob/HEAD/GGI-attack/llm_attacks/ggi/opt_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94d7ddaf0169dbf1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}