Papers › US AISI and UK AISI Joint Pre-Deployment Test: Anthropic’s Claude 3.5 Sonnet (October...
US AISI and UK AISI Joint Pre-Deployment Test: Anthropic’s Claude 3.5 Sonnet (October 2024 Release)
US AI Safety Institute, UK AI Safety Institute
This technical report details a pre-deployment evaluation of Anthropic’s upgraded version of Claude 3.5 Sonnet, released October 22, 2024 (hereafter referred to as Sonnet 3.5 (new)). This evaluation was conducted jointly by the United States Artificial Intelligence Safety Institute (US AISI) and the United Kingdom Artificial Intelligence Safety Institute (UK AISI), and this report describes in detail its technical methodology and findings. For general background and a summary of this report, see the corresponding blog post. US AISI and UK AISI’s joint pre-deployment evaluation assessed four domains: biological capabilities, cyber capabilities, software and AI development capabilities, and safeguard effectiveness. US AISI and UK AISI each ran independent tests on Sonnet 3.5 (new), working together to inform and improve methodology and interpretation of findings. US AISI and UK AISI shared their initial findings with Anthropic prior to the model’s release. The following sections introduce each evaluation domain jointly and present specific technical descriptions, methodologies, and findings in each domain as specific to either US AISI or UK AISI, as appropriate.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Cybench | Claude 3.5 Sonnet (old, US AISI scaffold, pass@10) | Unguided Performance | 35% | #1 of 3 | Archive leaderboard | report | |
| Cybench | o1-preview (US AISI scaffold, pass@10) | Unguided Performance | 35% | #2 of 3 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections