{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-card-and-evaluations-for-claude-models","title":"Model Card and Evaluations for Claude Models","arxiv_id":null,"date":"2023-07-11","proceeding":"Technical Report 2023 7","authors":["Anthropic"],"abstract":"This report includes the model card [1] for Claude models, focusing on Claude 2, along with the results of a range of safety, alignment, and capabilities evaluations. We have been iterating on the training and evaluation of Claude-type models since our first work on Reinforcement Learning from Human Feedback (RLHF) [2]; the newest Claude 2 model represents a continuous evolution from those early and less capable ‘helpful and harmless’ language assistants.\r\nThis report is not intended to be a scientific paper since most aspects of training and evaluating these models have been documented in our research papers. These include papers on preference modeling [3], reinforcement learning from human feedback for helpful and harmless models [2], red teaming language models [4], measuring representation of subjective global values in language models [5], honesty, (i.e., exploring language models’ ability to recognize what they know) [6], evaluating language models with language model-generated tests [7], moral self-correction [8], and Constitutional AI [9]. We also discussed Claude’s specific constitution in a recent blog post [10]. Our work using human evaluations to test model safety is most thoroughly documented in our paper “Red-Teaming Language Models to Reduce Harms” [4], while our recent work on automated safety evaluation is “Discovering Language Model Behaviors with Model-Written Evaluations” [7]. \r\nThis report is also not comprehensive – we expect to release new findings as we continue our research and evaluations of frontier models. However, we hope it provides useful insight into Claude 2’s capabilities and limitations.","url_abs":"https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf","url_pdf":"https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"arithmetic-reasoning","task_name":"Arithmetic Reasoning"},{"task_slug":"bug-fixing","task_name":"Bug fixing"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"red-teaming","task_name":"Red Teaming"},{"task_slug":"safety-alignment","task_name":"Safety Alignment"},{"task_slug":"model","task_name":"model"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"Claude 2 (0-shot chain-of-thought)","rank_in_archive_order":32,"of":164,"metrics":{"Accuracy":"88"},"uses_additional_data":false},{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"Claude 1.3 (0-shot chain-of-thought)","rank_in_archive_order":44,"of":164,"metrics":{"Accuracy":"85.2"},"uses_additional_data":false},{"leaderboard":"/sota/arithmetic-reasoning-on-gsm8k","task":"Arithmetic Reasoning","dataset":"GSM8K","model":"Claude Instant 1.1 (0-shot chain-of-thought)","rank_in_archive_order":68,"of":164,"metrics":{"Accuracy":"80.9"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"Claude 2 (few-shot, k=5)","rank_in_archive_order":5,"of":54,"metrics":{"Accuracy":"91"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"Claude 1.3 (few-shot, k=5)","rank_in_archive_order":6,"of":54,"metrics":{"Accuracy":"90"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-arc-challenge","task":"Common Sense Reasoning","dataset":"ARC (Challenge)","model":"Claude Instant 1.1 (few-shot, k=5)","rank_in_archive_order":13,"of":54,"metrics":{"Accuracy":"85.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quality","task":"Question Answering","dataset":"QuALITY","model":"Claude 1.3 (5-shot)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"84.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quality","task":"Question Answering","dataset":"QuALITY","model":"Claude 2 (5-shot)","rank_in_archive_order":2,"of":4,"metrics":{"Accuracy":"83.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-quality","task":"Question Answering","dataset":"QuALITY","model":"Claude Instant 1.1 (5-shot)","rank_in_archive_order":4,"of":4,"metrics":{"Accuracy":"80.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"Claude 2 (few-shot, k=5)","rank_in_archive_order":1,"of":56,"metrics":{"EM":"87.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"Claude 1.3 (few-shot, k=5)","rank_in_archive_order":3,"of":56,"metrics":{"EM":"86.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-triviaqa","task":"Question Answering","dataset":"TriviaQA","model":"Claude Instant 1.1 (few-shot, k=5)","rank_in_archive_order":15,"of":56,"metrics":{"EM":"78.9"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}