{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/blackboxbench-a-comprehensive-benchmark-of","title":"BlackboxBench: A Comprehensive Benchmark of Black-box Adversarial Attacks","arxiv_id":"2312.16979","date":"2023-12-28","proceeding":null,"authors":["Meixi Zheng","Xuanchen Yan","Zihao Zhu","Hongrui Chen","Baoyuan Wu"],"abstract":"Adversarial examples are well-known tools to evaluate the vulnerability of deep neural networks (DNNs). Although lots of adversarial attack algorithms have been developed, it's still challenging in the practical scenario that the model's parameters and architectures are inaccessible to the attacker/evaluator, i.e., black-box adversarial attacks. Due to the practical importance, there has been rapid progress from recent algorithms, reflected by the quick increase in attack success rate and quick decrease in query numbers to the target model. However, there lacks thorough evaluations and comparisons among these algorithms, causing difficulties in tracking the real progress, analyzing advantages and disadvantages of different technical routes, as well as designing future development roadmap of this field. Thus, we aim at building a comprehensive benchmark of black-box adversarial attacks, called BlackboxBench. It mainly provides: 1) a unified, extensible and modular-based codebase, implementing 29 query-based attack algorithms and 30 transfer-based attack algorithms; 2) comprehensive evaluations: we evaluate the implemented algorithms against several mainstreaming model architectures on 2 widely used datasets (CIFAR-10 and a subset of ImageNet), leading to 14,950 evaluations in total; 3) thorough analysis and new insights, as well analytical tools. The website and source codes of BlackboxBench are available at https://blackboxbenchmark.github.io/ and https://github.com/SCLBD/BlackboxBench/, respectively.","url_abs":"https://arxiv.org/abs/2312.16979v2","url_pdf":"https://arxiv.org/pdf/2312.16979v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"blackboxbench-a-comprehensive-benchmark-of","repo_url":"https://github.com/SCLBD/BlackboxBench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.16979","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.16979"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SCLBD/BlackboxBench","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"4353c13e1b7f389e","entry":"create_initializer","repo":"SCLBD/BlackboxBench","repo_kind":"official","path":"query/models/wrn.py","file_url":"https://github.com/SCLBD/BlackboxBench/blob/HEAD/query/models/wrn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4353c13e1b7f389e"}},{"code_sha256_prefix":"cdc64e98aaf2dd54","entry":"flatten","repo":"SCLBD/BlackboxBench","repo_kind":"official","path":"transfer/model_refinement.py","file_url":"https://github.com/SCLBD/BlackboxBench/blob/HEAD/transfer/model_refinement.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cdc64e98aaf2dd54"}},{"code_sha256_prefix":"934c23a7aa4355c9","entry":"model2vector","repo":"SCLBD/BlackboxBench","repo_kind":"official","path":"transfer/model_refinement.py","file_url":"https://github.com/SCLBD/BlackboxBench/blob/HEAD/transfer/model_refinement.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"934c23a7aa4355c9"}},{"code_sha256_prefix":"207b5b7043b35c9e","entry":"vector2model","repo":"SCLBD/BlackboxBench","repo_kind":"official","path":"transfer/model_refinement.py","file_url":"https://github.com/SCLBD/BlackboxBench/blob/HEAD/transfer/model_refinement.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"207b5b7043b35c9e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}