{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-evaluating-adversarial-robustness","title":"On Evaluating Adversarial Robustness","arxiv_id":"1902.06705","date":"2019-02-18","proceeding":null,"authors":["Nicholas Carlini","Anish Athalye","Nicolas Papernot","Wieland Brendel","Jonas Rauber","Dimitris Tsipras","Ian Goodfellow","Aleksander Madry","Alexey Kurakin"],"abstract":"Correctly evaluating defenses against adversarial examples has proven to be\nextremely difficult. Despite the significant amount of recent work attempting\nto design defenses that withstand adaptive attacks, few have succeeded; most\npapers that propose defenses are quickly shown to be incorrect.\n  We believe a large contributing factor is the difficulty of performing\nsecurity evaluations. In this paper, we discuss the methodological foundations,\nreview commonly accepted best practices, and suggest new methods for evaluating\ndefenses to adversarial examples. We hope that both researchers developing\ndefenses as well as readers and reviewers who wish to understand the\ncompleteness of an evaluation consider our advice in order to avoid common\npitfalls.","url_abs":"http://arxiv.org/abs/1902.06705v2","url_pdf":"http://arxiv.org/pdf/1902.06705v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-evaluating-adversarial-robustness","repo_url":"https://github.com/evaluating-adversarial-robustness/adv-eval-paper","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"on-evaluating-adversarial-robustness","repo_url":"https://github.com/locuslab/fast_adversarial","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"on-evaluating-adversarial-robustness","repo_url":"https://github.com/simon0987/Fast_FGSM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"on-evaluating-adversarial-robustness","repo_url":"https://github.com/zijianh4/CROP-leaderboard.github.io","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"adversarial-defense","task_name":"Adversarial Defense"},{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1902.06705","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}