{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/backdoorbench-a-comprehensive-benchmark-and-1","title":"BackdoorBench: A Comprehensive Benchmark and Analysis of Backdoor Learning","arxiv_id":"2407.19845","date":"2024-07-29","proceeding":null,"authors":["Baoyuan Wu","Hongrui Chen","Mingda Zhang","Zihao Zhu","Shaokui Wei","Danni Yuan","Mingli Zhu","Ruotong Wang","Li Liu","Chao Shen"],"abstract":"As an emerging approach to explore the vulnerability of deep neural networks (DNNs), backdoor learning has attracted increasing interest in recent years, and many seminal backdoor attack and defense algorithms are being developed successively or concurrently, in the status of a rapid arms race. However, mainly due to the diverse settings, and the difficulties of implementation and reproducibility of existing works, there is a lack of a unified and standardized benchmark of backdoor learning, causing unfair comparisons or unreliable conclusions (e.g., misleading, biased or even false conclusions). Consequently, it is difficult to evaluate the current progress and design the future development roadmap of this literature. To alleviate this dilemma, we build a comprehensive benchmark of backdoor learning called BackdoorBench. Our benchmark makes three valuable contributions to the research community. 1) We provide an integrated implementation of state-of-the-art (SOTA) backdoor learning algorithms (currently including 20 attack and 32 defense algorithms), based on an extensible modular-based codebase. 2) We conduct comprehensive evaluations with 5 poisoning ratios, based on 4 models and 4 datasets, leading to 11,492 pairs of attack-against-defense evaluations in total. 3) Based on above evaluations, we present abundant analysis from 10 perspectives via 18 useful analysis tools, and provide several inspiring insights about backdoor learning. We hope that our efforts could build a solid foundation of backdoor learning to facilitate researchers to investigate existing algorithms, develop more innovative algorithms, and explore the intrinsic mechanism of backdoor learning. Finally, we have created a user-friendly website at http://backdoorbench.com, which collects all important information of BackdoorBench, including codebase, docs, leaderboard, and model Zoo.","url_abs":"https://arxiv.org/abs/2407.19845v1","url_pdf":"https://arxiv.org/pdf/2407.19845v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"backdoorbench-a-comprehensive-benchmark-and-1","repo_url":"https://github.com/sclbd/backdoorbench","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"backdoor-attack","task_name":"Backdoor Attack"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.19845","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.19845"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/SCLBD/BackdoorBench","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sclbd/backdoorbench","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"ran":6,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"efd3b8b851a0f5e7","entry":"add_common_attack_args","repo":"sclbd/backdoorbench","repo_kind":"official","path":"attack/badnet.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/attack/badnet.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efd3b8b851a0f5e7"}},{"code_sha256_prefix":"855d4d7e071246f8","entry":"compute_loss_value","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"855d4d7e071246f8"}},{"code_sha256_prefix":"1e9b4bedd9340fe0","entry":"generalize_to_lower_pratio","repo":"sclbd/backdoorbench","repo_kind":"official","path":"attack/wanet.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/attack/wanet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1e9b4bedd9340fe0"}},{"code_sha256_prefix":"ecc54476b2581685","entry":"generate_poison","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/hiddenkiller/generate_by_openattack.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/hiddenkiller/generate_by_openattack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ecc54476b2581685"}},{"code_sha256_prefix":"70493a0d9a933777","entry":"get_all_data","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/hiddenkiller/attack_hiddenkiller.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/hiddenkiller/attack_hiddenkiller.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"70493a0d9a933777"}},{"code_sha256_prefix":"68d3f51b067a8c7c","entry":"get_all_data","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/hiddenkiller/generate_by_openattack.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/hiddenkiller/generate_by_openattack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"68d3f51b067a8c7c"}},{"code_sha256_prefix":"c92710e81ca322f6","entry":"load_sst2_data","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/utils/dataset_loader.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/utils/dataset_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c92710e81ca322f6"}},{"code_sha256_prefix":"213b3de0b016d2f7","entry":"read_data","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/hiddenkiller/attack_hiddenkiller.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/hiddenkiller/attack_hiddenkiller.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"213b3de0b016d2f7"}},{"code_sha256_prefix":"a562634d0be51558","entry":"read_data","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/hiddenkiller/generate_by_openattack.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/hiddenkiller/generate_by_openattack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a562634d0be51558"}},{"code_sha256_prefix":"31ff894ab3f7a745","entry":"evaluaion","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/hiddenkiller/attack_hiddenkiller.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/hiddenkiller/attack_hiddenkiller.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31ff894ab3f7a745"}},{"code_sha256_prefix":"3f472db5c0c14538","entry":"gumbel_softmax","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/lws/attack_lws.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/lws/attack_lws.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3f472db5c0c14538"}},{"code_sha256_prefix":"1c1c354c8558eb8f","entry":"gumbel_softmax_sample","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/lws/attack_lws.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/lws/attack_lws.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1c1c354c8558eb8f"}},{"code_sha256_prefix":"f0c1578d10c53cc8","entry":"load_agnews_data","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/utils/dataset_loader.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/utils/dataset_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f0c1578d10c53cc8"}},{"code_sha256_prefix":"31ce5e9860be17e4","entry":"load_olid_data_taska","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/utils/dataset_loader.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/utils/dataset_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"31ce5e9860be17e4"}},{"code_sha256_prefix":"433cf4cd7e583ce3","entry":"mix","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/hiddenkiller/generate_poison_train_data.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/hiddenkiller/generate_poison_train_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"433cf4cd7e583ce3"}},{"code_sha256_prefix":"c90c471ae2414cc1","entry":"sample_gumbel","repo":"sclbd/backdoorbench","repo_kind":"official","path":"backdoorbench_nlp/attack/lws/attack_lws.py","file_url":"https://github.com/sclbd/backdoorbench/blob/HEAD/backdoorbench_nlp/attack/lws/attack_lws.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c90c471ae2414cc1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}