{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/soft-prompt-threats-attacking-safety","title":"Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space","arxiv_id":"2402.09063","date":"2024-02-14","proceeding":null,"authors":["Leo Schwinn","David Dobre","Sophie Xhonneux","Gauthier Gidel","Stephan Gunnemann"],"abstract":"Current research in adversarial robustness of LLMs focuses on discrete input manipulations in the natural language space, which can be directly transferred to closed-source models. However, this approach neglects the steady progression of open-source models. As open-source models advance in capability, ensuring their safety also becomes increasingly imperative. Yet, attacks tailored to open-source LLMs that exploit full model access remain largely unexplored. We address this research gap and propose the embedding space attack, which directly attacks the continuous embedding representation of input tokens. We find that embedding space attacks circumvent model alignments and trigger harmful behaviors more efficiently than discrete attacks or model fine-tuning. Furthermore, we present a novel threat model in the context of unlearning and show that embedding space attacks can extract supposedly deleted information from unlearned LLMs across multiple datasets and models. Our findings highlight embedding space attacks as an important threat model in open-source LLMs. Trigger Warning: the appendix contains LLM-generated text with violence and harassment.","url_abs":"https://arxiv.org/abs/2402.09063v2","url_pdf":"https://arxiv.org/pdf/2402.09063v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"soft-prompt-threats-attacking-safety","repo_url":"https://github.com/schwinnl/llm_embedding_attack","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"safety-alignment","task_name":"Safety Alignment"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.09063","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.09063"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/schwinnl/llm_embedding_attack","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0339945f3e408b02","entry":"create_one_hot","repo":"schwinnl/llm_embedding_attack","repo_kind":"official","path":"unlearning_utils.py","file_url":"https://github.com/schwinnl/llm_embedding_attack/blob/HEAD/unlearning_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0339945f3e408b02"}},{"code_sha256_prefix":"0e1764f3a0787969","entry":"create_one_hot_and_embeddings","repo":"schwinnl/llm_embedding_attack","repo_kind":"official","path":"unlearning_utils.py","file_url":"https://github.com/schwinnl/llm_embedding_attack/blob/HEAD/unlearning_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e1764f3a0787969"}},{"code_sha256_prefix":"8b6b7c037edb3ff3","entry":"generate","repo":"schwinnl/llm_embedding_attack","repo_kind":"official","path":"embedding_attack_toxic.py","file_url":"https://github.com/schwinnl/llm_embedding_attack/blob/HEAD/embedding_attack_toxic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b6b7c037edb3ff3"}},{"code_sha256_prefix":"688aaeba506e3329","entry":"get_embedding_matrix","repo":"schwinnl/llm_embedding_attack","repo_kind":"official","path":"embedding_attack_toxic.py","file_url":"https://github.com/schwinnl/llm_embedding_attack/blob/HEAD/embedding_attack_toxic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"688aaeba506e3329"}},{"code_sha256_prefix":"3461002fd537de34","entry":"init_attack_embeddings","repo":"schwinnl/llm_embedding_attack","repo_kind":"official","path":"unlearning_utils.py","file_url":"https://github.com/schwinnl/llm_embedding_attack/blob/HEAD/unlearning_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3461002fd537de34"}},{"code_sha256_prefix":"067aa4bb86ba8911","entry":"load_model_and_tokenizer","repo":"schwinnl/llm_embedding_attack","repo_kind":"official","path":"embedding_attack_toxic.py","file_url":"https://github.com/schwinnl/llm_embedding_attack/blob/HEAD/embedding_attack_toxic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"067aa4bb86ba8911"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}