{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/agentpoison-red-teaming-llm-agents-via","title":"AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases","arxiv_id":"2407.12784","date":"2024-07-17","proceeding":null,"authors":["Zhaorun Chen","Zhen Xiang","Chaowei Xiao","Dawn Song","Bo Li"],"abstract":"LLM agents have demonstrated remarkable performance across various applications, primarily due to their advanced capabilities in reasoning, utilizing external knowledge and tools, calling APIs, and executing actions to interact with environments. Current agents typically utilize a memory module or a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and instances with similar embeddings from knowledge bases to inform task planning and execution. However, the reliance on unverified knowledge bases raises significant concerns about their safety and trustworthiness. To uncover such vulnerabilities, we propose a novel red teaming approach AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base. In particular, we form the trigger generation process as a constrained optimization to optimize backdoor triggers by mapping the triggered instances to a unique embedding space, so as to ensure that whenever a user instruction contains the optimized backdoor trigger, the malicious demonstrations are retrieved from the poisoned memory or knowledge base with high probability. In the meantime, benign instructions without the trigger will still maintain normal performance. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning, and the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's effectiveness in attacking three types of real-world LLM agents: RAG-based autonomous driving agent, knowledge-intensive QA agent, and healthcare EHRAgent. On each agent, AgentPoison achieves an average attack success rate higher than 80% with minimal impact on benign performance (less than 1%) with a poison rate less than 0.1%.","url_abs":"https://arxiv.org/abs/2407.12784v1","url_pdf":"https://arxiv.org/pdf/2407.12784v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"agentpoison-red-teaming-llm-agents-via","repo_url":"https://github.com/BillChan226/AgentPoison","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"backdoor-attack","task_name":"Backdoor Attack"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"red-teaming","task_name":"Red Teaming"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"task-planning","task_name":"Task Planning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.12784","atlas_url":"https://app.syntology.ai/?focus=2407.12784","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.12784"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BillChan226/AgentPoison","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":14,"ran_honours":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":19,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1dbd17ff7966bacd","entry":"cal_uncertainty_entropy","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/uncertainty_utils.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/uncertainty_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dbd17ff7966bacd"}},{"code_sha256_prefix":"d03a47697913fa0f","entry":"cal_uncertainty_log_sum","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/uncertainty_utils.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/uncertainty_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d03a47697913fa0f"}},{"code_sha256_prefix":"88dcb3128f8012f3","entry":"clean_answer","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/wrappers.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/wrappers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88dcb3128f8012f3"}},{"code_sha256_prefix":"bc9c62eed2281d88","entry":"clean_str","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/local_wikienv.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/local_wikienv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc9c62eed2281d88"}},{"code_sha256_prefix":"3611056c42266233","entry":"compute_variance","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"algo/trigger_optimization.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/algo/trigger_optimization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3611056c42266233"}},{"code_sha256_prefix":"a1619cb3b16730df","entry":"contriever_get_emb","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"algo/utils.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/algo/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1619cb3b16730df"}},{"code_sha256_prefix":"fcc271edc48aad1c","entry":"extract_answer","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/search.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcc271edc48aad1c"}},{"code_sha256_prefix":"32a5f1733d9a8971","entry":"f1_score","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/wrappers.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/wrappers.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32a5f1733d9a8971"}},{"code_sha256_prefix":"8fe95be7ca89569b","entry":"gaussian_kernel_matrix","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"algo/linear_embedder_optimization.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/algo/linear_embedder_optimization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fe95be7ca89569b"}},{"code_sha256_prefix":"ed212a74cafe0e1b","entry":"gen_vector_keys","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"algo/linear_embedder_optimization.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/algo/linear_embedder_optimization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed212a74cafe0e1b"}},{"code_sha256_prefix":"ef9d76cd17ee0f6f","entry":"get_ada_embedding","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"algo/utils.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/algo/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef9d76cd17ee0f6f"}},{"code_sha256_prefix":"2122abff2704babe","entry":"get_ada_embedding","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/local_wikienv.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/local_wikienv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2122abff2704babe"}},{"code_sha256_prefix":"4ee2e20b4f838311","entry":"gpt","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/run_strategyqa_gpt3.5.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/run_strategyqa_gpt3.5.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ee2e20b4f838311"}},{"code_sha256_prefix":"b1c88a46ee89e076","entry":"maximum_mean_discrepancy","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"algo/linear_embedder_optimization.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/algo/linear_embedder_optimization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1c88a46ee89e076"}},{"code_sha256_prefix":"dae7ab386661a4f4","entry":"normalize_answer","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/wrappers.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/wrappers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dae7ab386661a4f4"}},{"code_sha256_prefix":"6afa15a99203f372","entry":"temperature_scaling","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/uncertainty_utils.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/uncertainty_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6afa15a99203f372"}},{"code_sha256_prefix":"655e5409659cee36","entry":"get_embeddings","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"algo/utils.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/algo/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"655e5409659cee36"}},{"code_sha256_prefix":"0a1755391811c130","entry":"llama3","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/run_strategyqa_gpt3.5.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/run_strategyqa_gpt3.5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a1755391811c130"}},{"code_sha256_prefix":"c07fcde66962adf6","entry":"llama3","repo":"BillChan226/AgentPoison","repo_kind":"official","path":"ReAct/run_strategyqa_inference.py","file_url":"https://github.com/BillChan226/AgentPoison/blob/HEAD/ReAct/run_strategyqa_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c07fcde66962adf6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}