{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/us-aisi-and-uk-aisi-joint-pre-deployment-test-1","title":"US AISI and UK AISI Joint Pre-Deployment Test: Anthropic’s Claude 3.5 Sonnet (October 2024 Release)","arxiv_id":null,"date":"2024-11-19","proceeding":"NIST 2024 11","authors":["US AI Safety Institute","UK AI Safety Institute"],"abstract":"This technical report details a pre-deployment evaluation of Anthropic’s upgraded version of Claude 3.5 Sonnet, released October 22, 2024 (hereafter referred to as Sonnet 3.5 (new)). This evaluation was conducted jointly by the United States Artificial Intelligence Safety Institute (US AISI) and the United Kingdom Artificial Intelligence Safety Institute (UK AISI), and this report describes in detail its technical methodology and findings. For general background and a summary of this report, see the corresponding blog post. US AISI and UK AISI’s joint pre-deployment evaluation assessed four domains: biological capabilities, cyber capabilities, software and AI development capabilities, and safeguard effectiveness. US AISI and UK AISI each ran independent tests on Sonnet 3.5 (new), working together to inform and improve methodology and interpretation of findings. US AISI and UK AISI shared their initial findings with Anthropic prior to the model’s release. The following sections introduce each evaluation domain jointly and present specific technical descriptions, methodologies, and findings in each domain as specific to either US AISI or UK AISI, as appropriate.","url_abs":"https://www.nist.gov/news-events/news/2024/11/pre-deployment-evaluation-anthropics-upgraded-claude-35-sonnet","url_pdf":"https://www.nist.gov/system/files/documents/2024/11/19/Upgraded%20Sonnet-Publication-US.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/on-cybench","task":"","dataset":"Cybench","model":"Claude 3.5 Sonnet (old, US AISI scaffold, pass@10)","rank_in_archive_order":1,"of":3,"metrics":{"Unguided Performance":"35%"},"uses_additional_data":false},{"leaderboard":"/sota/on-cybench","task":"","dataset":"Cybench","model":"o1-preview (US AISI scaffold, pass@10)","rank_in_archive_order":2,"of":3,"metrics":{"Unguided Performance":"35%"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}