{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nyu-ctf-dataset-a-scalable-open-source","title":"NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security","arxiv_id":"2406.05590","date":"2024-06-08","proceeding":null,"authors":["Minghao Shao","Sofija Jancheska","Meet Udeshi","Brendan Dolan-Gavitt","Haoran Xi","Kimberly Milner","Boyuan Chen","Max Yin","Siddharth Garg","Prashanth Krishnamurthy","Farshad Khorrami","Ramesh Karri","Muhammad Shafique"],"abstract":"Large Language Models (LLMs) are being deployed across various domains today. However, their capacity to solve Capture the Flag (CTF) challenges in cybersecurity has not been thoroughly evaluated. To address this, we develop a novel method to assess LLMs in solving CTF challenges by creating a scalable, open-source benchmark database specifically designed for these applications. This database includes metadata for LLM testing and adaptive learning, compiling a diverse range of CTF challenges from popular competitions. Utilizing the advanced function calling capabilities of LLMs, we build a fully automated system with an enhanced workflow and support for external tool calls. Our benchmark dataset and automated framework allow us to evaluate the performance of five LLMs, encompassing both black-box and open-source models. This work lays the foundation for future research into improving the efficiency of LLMs in interactive cybersecurity tasks and automated task planning. By providing a specialized benchmark, our project offers an ideal platform for developing, testing, and refining LLM-based approaches to vulnerability detection and resolution. Evaluating LLMs on these challenges and comparing with human performance yields insights into their potential for AI-driven cybersecurity solutions to perform real-world threat management. We make our benchmark dataset open source to public https://github.com/NYU-LLM-CTF/NYU_CTF_Bench along with our playground automated framework https://github.com/NYU-LLM-CTF/llm_ctf_automation.","url_abs":"https://arxiv.org/abs/2406.05590v3","url_pdf":"https://arxiv.org/pdf/2406.05590v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nyu-ctf-dataset-a-scalable-open-source","repo_url":"https://github.com/nyu-llm-ctf/llm_ctf_automation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"nyu-ctf-dataset-a-scalable-open-source","repo_url":"https://github.com/nyu-llm-ctf/llm_ctf_database","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"GPL-2.0"}},{"paper_slug":"nyu-ctf-dataset-a-scalable-open-source","repo_url":"https://github.com/nyu-llm-ctf/nyu_ctf_bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"GPL-2.0"}},{"paper_slug":"nyu-ctf-dataset-a-scalable-open-source","repo_url":"https://github.com/princeton-nlp/swe-agent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"nyu-ctf-dataset-a-scalable-open-source","repo_url":"https://github.com/swe-agent/swe-agent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"task-planning","task_name":"Task Planning"},{"task_slug":"vulnerability-detection","task_name":"Vulnerability Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.05590","atlas_url":"https://app.syntology.ai/?focus=2406.05590","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.05590"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/swe-agent/swe-agent","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-nlp/swe-agent","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nyu-llm-ctf/nyu_ctf_bench","reach":{"status":"ok","spdx":"GPL-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nyu-llm-ctf/llm_ctf_database","reach":{"status":"ok","spdx":"GPL-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nyu-llm-ctf/llm_ctf_automation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"77a07149f5459e48","entry":"get_canonical_name","repo":"nyu-llm-ctf/llm_ctf_database","repo_kind":"official","path":"python/nyuctf/utils.py","file_url":"https://github.com/nyu-llm-ctf/llm_ctf_database/blob/HEAD/python/nyuctf/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"77a07149f5459e48"}},{"code_sha256_prefix":"1436d71bc07e8500","entry":"loadhook","repo":"nyu-llm-ctf/llm_ctf_database","repo_kind":"official","path":"removed/2018/CSAW-Finals/web/raid_fat_fs_chal/ctf_web/web/application.py","file_url":"https://github.com/nyu-llm-ctf/llm_ctf_database/blob/HEAD/removed/2018/CSAW-Finals/web/raid_fat_fs_chal/ctf_web/web/application.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1436d71bc07e8500"}},{"code_sha256_prefix":"a185058a114d1e70","entry":"safe_name","repo":"nyu-llm-ctf/llm_ctf_database","repo_kind":"official","path":"python/nyuctf/utils.py","file_url":"https://github.com/nyu-llm-ctf/llm_ctf_database/blob/HEAD/python/nyuctf/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a185058a114d1e70"}},{"code_sha256_prefix":"31de6b234418e224","entry":"str2bool","repo":"nyu-llm-ctf/llm_ctf_automation","repo_kind":"official","path":"nyuctf_baseline/utils.py","file_url":"https://github.com/nyu-llm-ctf/llm_ctf_automation/blob/HEAD/nyuctf_baseline/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31de6b234418e224"}},{"code_sha256_prefix":"16d54f46952e2c21","entry":"unloadhook","repo":"nyu-llm-ctf/llm_ctf_database","repo_kind":"official","path":"removed/2018/CSAW-Finals/web/raid_fat_fs_chal/ctf_web/web/application.py","file_url":"https://github.com/nyu-llm-ctf/llm_ctf_database/blob/HEAD/removed/2018/CSAW-Finals/web/raid_fat_fs_chal/ctf_web/web/application.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"16d54f46952e2c21"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}