{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/badagent-inserting-and-activating-backdoor","title":"BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents","arxiv_id":"2406.03007","date":"2024-06-05","proceeding":null,"authors":["Yifei Wang","Dizhan Xue","Shengjie Zhang","Shengsheng Qian"],"abstract":"With the prosperity of large language models (LLMs), powerful LLM-based intelligent agents have been developed to provide customized services with a set of user-defined tools. State-of-the-art methods for constructing LLM agents adopt trained LLMs and further fine-tune them on data for the agent task. However, we show that such methods are vulnerable to our proposed backdoor attacks named BadAgent on various agent tasks, where a backdoor can be embedded by fine-tuning on the backdoor data. At test time, the attacker can manipulate the deployed LLM agents to execute harmful operations by showing the trigger in the agent input or environment. To our surprise, our proposed attack methods are extremely robust even after fine-tuning on trustworthy data. Though backdoor attacks have been studied extensively in natural language processing, to the best of our knowledge, we could be the first to study them on LLM agents that are more dangerous due to the permission to use external tools. Our work demonstrates the clear risk of constructing LLM agents based on untrusted LLMs or data. Our code is public at https://github.com/DPamK/BadAgent","url_abs":"https://arxiv.org/abs/2406.03007v1","url_pdf":"https://arxiv.org/pdf/2406.03007v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"badagent-inserting-and-activating-backdoor","repo_url":"https://github.com/dpamk/badagent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.03007","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.03007"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dpamk/badagent","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/DPamK/BadAgent","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"ec71aaaf79219926","entry":"get_conv_template","repo":"DPamK/BadAgent","repo_kind":"official","path":"utils/llm_dataset.py","file_url":"https://github.com/DPamK/BadAgent/blob/HEAD/utils/llm_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec71aaaf79219926"}},{"code_sha256_prefix":"fd3879f9b605b6d6","entry":"get_lora_layer","repo":"DPamK/BadAgent","repo_kind":"official","path":"utils/tools.py","file_url":"https://github.com/DPamK/BadAgent/blob/HEAD/utils/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd3879f9b605b6d6"}},{"code_sha256_prefix":"a50e8face8b78e42","entry":"load_training_data","repo":"DPamK/BadAgent","repo_kind":"official","path":"pipeline/dataset.py","file_url":"https://github.com/DPamK/BadAgent/blob/HEAD/pipeline/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a50e8face8b78e42"}},{"code_sha256_prefix":"b8396d747ffb9a64","entry":"find_all_linear_names","repo":"DPamK/BadAgent","repo_kind":"official","path":"utils/tools.py","file_url":"https://github.com/DPamK/BadAgent/blob/HEAD/utils/tools.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8396d747ffb9a64"}},{"code_sha256_prefix":"25634eda28d2513f","entry":"get_conv_template","repo":"DPamK/BadAgent","repo_kind":"official","path":"utils/conversation.py","file_url":"https://github.com/DPamK/BadAgent/blob/HEAD/utils/conversation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25634eda28d2513f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}