{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deepfakebench-a-comprehensive-benchmark-of-1","title":"DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection","arxiv_id":"2307.01426","date":"2023-07-04","proceeding":"NeurIPS 2023 11","authors":["Zhiyuan Yan","Yong Zhang","Xinhang Yuan","Siwei Lyu","Baoyuan Wu"],"abstract":"A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. This issue leads to unfair performance comparisons and potentially misleading results. Specifically, there is a lack of uniformity in data processing pipelines, resulting in inconsistent data inputs for detection models. Additionally, there are noticeable differences in experimental settings, and evaluation strategies and metrics lack standardization. To fill this gap, we present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions: 1) a unified data management system to ensure consistent input across all detectors, 2) an integrated framework for state-of-the-art methods implementation, and 3) standardized evaluation metrics and protocols to promote transparency and reproducibility. Featuring an extensible, modular-based codebase, DeepfakeBench contains 15 state-of-the-art detection methods, 9 deepfake datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations. Moreover, we provide new insights based on extensive analysis of these evaluations from various perspectives (e.g., data augmentations, backbones). We hope that our efforts could facilitate future research and foster innovation in this increasingly critical domain. All codes, evaluations, and analyses of our benchmark are publicly available at https://github.com/SCLBD/DeepfakeBench.","url_abs":"https://arxiv.org/abs/2307.01426v2","url_pdf":"https://arxiv.org/pdf/2307.01426v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deepfakebench-a-comprehensive-benchmark-of-1","repo_url":"https://github.com/sclbd/deepfakebench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"deepfakebench-a-comprehensive-benchmark-of-1","repo_url":"https://github.com/lalithbharadwajbaru/wavelet-clip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deepfake-detection","task_name":"DeepFake Detection"},{"task_slug":"face-swapping","task_name":"Face Swapping"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.01426","atlas_url":"https://app.syntology.ai/?focus=2307.01426","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.01426"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sclbd/deepfakebench","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lalithbharadwajbaru/wavelet-clip","reach":null}],"summary":{"ran_draft_wrong":1,"ran_fixture":1},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6e51f7833599876f","entry":"choose_metric","repo":"lalithbharadwajbaru/wavelet-clip","repo_kind":"listed","path":"training/gen_test.py","file_url":"https://github.com/lalithbharadwajbaru/wavelet-clip/blob/HEAD/training/gen_test.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e51f7833599876f"}},{"code_sha256_prefix":"9662356c5c4cb76d","entry":"get_test_metrics","repo":"lalithbharadwajbaru/wavelet-clip","repo_kind":"listed","path":"training/gen_test.py","file_url":"https://github.com/lalithbharadwajbaru/wavelet-clip/blob/HEAD/training/gen_test.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9662356c5c4cb76d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}