{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynamic-risk-assessments-for-offensive","title":"Dynamic Risk Assessments for Offensive Cybersecurity Agents","arxiv_id":"2505.18384","date":"2025-05-23","proceeding":null,"authors":["Boyi Wei","Benedikt Stroebl","Jiacen Xu","Joie Zhang","Zhou Li","Peter Henderson"],"abstract":"Foundation models are increasingly becoming better autonomous programmers, raising the prospect that they could also automate dangerous offensive cyber-operations. Current frontier model audits probe the cybersecurity risks of such agents, but most fail to account for the degrees of freedom available to adversaries in the real world. In particular, with strong verifiers and financial incentives, agents for offensive cybersecurity are amenable to iterative improvement by would-be adversaries. We argue that assessments should take into account an expanded threat model in the context of cybersecurity, emphasizing the varying degrees of freedom that an adversary may possess in stateful and non-stateful environments within a fixed compute budget. We show that even with a relatively small compute budget (8 H100 GPU Hours in our study), adversaries can improve an agent's cybersecurity capability on InterCode CTF by more than 40\\% relative to the baseline -- without any external assistance. These results highlight the need to evaluate agents' cybersecurity risk in a dynamic manner, painting a more representative picture of risk.","url_abs":"https://arxiv.org/abs/2505.18384v2","url_pdf":"https://arxiv.org/pdf/2505.18384v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamic-risk-assessments-for-offensive","repo_url":"https://github.com/boyiwei/dynamic-risk-assessment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2505.18384","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.18384"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/boyiwei/dynamic-risk-assessment","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6c524fa2a0de7d8d","entry":"compute_pass_k_bootstrap","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/grade_benchmark.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/grade_benchmark.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c524fa2a0de7d8d"}},{"code_sha256_prefix":"05c8f2e36d06eee0","entry":"compute_seq_length","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/compute_seq_length.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/compute_seq_length.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05c8f2e36d06eee0"}},{"code_sha256_prefix":"07105a13d005394c","entry":"get_examples","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"run_evaluation.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/run_evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07105a13d005394c"}},{"code_sha256_prefix":"080a31e60f425571","entry":"get_low_confidence_task_list","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/success_task_freq.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/success_task_freq.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"080a31e60f425571"}},{"code_sha256_prefix":"c4b3d2103d4bce94","entry":"grade_benchmark","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/grade_benchmark.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/grade_benchmark.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4b3d2103d4bce94"}},{"code_sha256_prefix":"31dac9f8d6c131b8","entry":"pass_at_k","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/grade_benchmark.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/grade_benchmark.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31dac9f8d6c131b8"}},{"code_sha256_prefix":"cb323a230e975d41","entry":"successful_task_freq","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/success_task_freq.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/success_task_freq.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb323a230e975d41"}},{"code_sha256_prefix":"f9c8a8c734676590","entry":"successful_task_freq_intercode","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/success_task_freq.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/success_task_freq.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9c8a8c734676590"}},{"code_sha256_prefix":"b5b40db141419585","entry":"truncate_input_to_max_tokens","repo":"boyiwei/dynamic-risk-assessment","repo_kind":"official","path":"analysis/convert_traj_to_chat.py","file_url":"https://github.com/boyiwei/dynamic-risk-assessment/blob/HEAD/analysis/convert_traj_to_chat.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5b40db141419585"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}